[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:26:52 dist-test-slave-31kt krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:26:52 dist-test-slave-31kt krb5kdc[399](info): set up 2 sockets
Sep 25 16:26:52 dist-test-slave-31kt krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 16:26:53.447036 392 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/test-user.keytab.
W20260925 16:26:53.661221 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.139s user 0.000s sys 0.002s
Sep 25 16:26:53 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260925 16:26:54.032030 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.332s user 0.005s sys 0.000s
I20260925 16:26:54.033193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37919
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:54.170502 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:54.170873 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:54.170923 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:54.175441 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:26:54.175532 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:54.175552 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:54.175570 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:26:54.175585 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:26:54.181892 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:54.183408 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:54.184805 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:54.195652 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:54.195742 412 server_base.cc:1034] running on GCE node
W20260925 16:26:54.195667 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:54.195667 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:54.196846 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:54.198709 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:54.200124 412 hybrid_clock.cc:648] HybridClock initialized: now 1790353614200057 us; error 101 us; skew 500 ppm
Sep 25 16:26:54 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:54.203676 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:26:54.204897 412 webserver.cc:460] Webserver started at http://127.0.98.62:34847/ using document root <none> and password file <none>
I20260925 16:26:54.205651 412 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:54.205725 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:54.206007 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:54.208665 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "edc60fce6bf3427aae6780a0fbff575f"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "d343f13c6d1db8a4e8ab78f573559897"
server_key_iv: "ea017ac0f7db57f61329137c579f1aa6"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.209400 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "edc60fce6bf3427aae6780a0fbff575f"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "d343f13c6d1db8a4e8ab78f573559897"
server_key_iv: "ea017ac0f7db57f61329137c579f1aa6"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.214721 412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:26:54.218565 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.220744 412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:26:54.220973 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "edc60fce6bf3427aae6780a0fbff575f"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "d343f13c6d1db8a4e8ab78f573559897"
server_key_iv: "ea017ac0f7db57f61329137c579f1aa6"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.221168 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:54.242671 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:54.247205 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:54.247473 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:54.257262 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37919
I20260925 16:26:54.257284 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37919 every 8 connection(s)
I20260925 16:26:54.258672 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:26:54.262616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 412
I20260925 16:26:54.262791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:26:54.263574 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:26:54.264453 392 external_mini_cluster.cc:1428] Setting key f969db164737928ec28152df597fb2bd
I20260925 16:26:54.272624 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f: Bootstrap starting.
I20260925 16:26:54.275954 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f: Neither blocks nor log segments found. Creating new log.
I20260925 16:26:54.277069 480 log.cc:824] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:26:54.280155 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f: No bootstrap required, opened a new log
I20260925 16:26:54.285746 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc60fce6bf3427aae6780a0fbff575f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37919 } }
I20260925 16:26:54.286226 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:26:54.286360 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc60fce6bf3427aae6780a0fbff575f, State: Initialized, Role: FOLLOWER
I20260925 16:26:54.287014 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc60fce6bf3427aae6780a0fbff575f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37919 } }
I20260925 16:26:54.287215 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:26:54.287302 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:26:54.287418 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:26:54 dist-test-slave-31kt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:54.289028 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc60fce6bf3427aae6780a0fbff575f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37919 } }
I20260925 16:26:54.289582 480 leader_election.cc:302] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edc60fce6bf3427aae6780a0fbff575f; no voters:
I20260925 16:26:54.290047 480 leader_election.cc:288] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:26:54.290272 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:26:54.290649 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [term 1 LEADER]: Becoming Leader. State: Replica: edc60fce6bf3427aae6780a0fbff575f, State: Running, Role: LEADER
I20260925 16:26:54.291268 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc60fce6bf3427aae6780a0fbff575f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37919 } }
I20260925 16:26:54.291831 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:26:54.293706 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc60fce6bf3427aae6780a0fbff575f. Latest consensus state: current_term: 1 leader_uuid: "edc60fce6bf3427aae6780a0fbff575f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc60fce6bf3427aae6780a0fbff575f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37919 } } }
I20260925 16:26:54.293946 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [sys.catalog]: This master's current role is: LEADER
I20260925 16:26:54.298519 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:54.269108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.62:37919)
0925 16:26:54.269796 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:54.269827 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:54.271702 (+ 1875us) server_negotiation.cc:407] Connection header received
0925 16:26:54.276214 (+ 4512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:54.276243 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:54.276807 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:54.277284 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:54.280378 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.281609 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.283206 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.283805 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.284018 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:54.289916 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:54.289927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:54.289953 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:54.290013 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:54.292539 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:54.293641 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:54.293651 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:54.293658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:54.293998 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:54.294531 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:54.294534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:54.294536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:54.295220 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:54.295483 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:54.296567 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0925 16:26:54.297196 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":213,"threads_started":1}
I20260925 16:26:54.294723 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc60fce6bf3427aae6780a0fbff575f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc60fce6bf3427aae6780a0fbff575f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37919 } } }
I20260925 16:26:54.299068 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f [sys.catalog]: This master's current role is: LEADER
W20260925 16:26:54.299398 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:26:54.299474 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:26:54.299577 499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:26:54.300638 499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:26:54.309844 499 catalog_manager.cc:1347] Generated new cluster ID: a10e1fccbb8a4003bbe1f977852d9e5b
I20260925 16:26:54.309937 499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:26:54.327263 499 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:26:54.329119 499 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:26:54.341064 499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edc60fce6bf3427aae6780a0fbff575f: Generated new TSK 0
I20260925 16:26:54.342370 499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:26:54.382288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37919
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:54.546284 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:54.546674 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:54.546725 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:54.551179 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:54.551272 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:54.551344 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:26:54.557721 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:54.559297 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:54.560644 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:54.572865 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:54.572945 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:54.572963 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:54.573263 506 server_base.cc:1034] running on GCE node
I20260925 16:26:54.573933 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:54.574779 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:54.576007 506 hybrid_clock.cc:648] HybridClock initialized: now 1790353614575967 us; error 52 us; skew 500 ppm
Sep 25 16:26:54 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:54.579268 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:26:54.580423 506 webserver.cc:460] Webserver started at http://127.0.98.1:38283/ using document root <none> and password file <none>
I20260925 16:26:54.581187 506 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:54.581280 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:54.581517 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:54.584484 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "badd2379120c484899492b5fed6700dd"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "54bcc3fd2a7d7c043d7a775f68008132"
server_key_iv: "30a3f536d259226507530cd26f45fa90"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.585371 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "badd2379120c484899492b5fed6700dd"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "54bcc3fd2a7d7c043d7a775f68008132"
server_key_iv: "30a3f536d259226507530cd26f45fa90"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.591044 506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:26:54.595077 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.596920 506 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260925 16:26:54.597111 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "badd2379120c484899492b5fed6700dd"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "54bcc3fd2a7d7c043d7a775f68008132"
server_key_iv: "30a3f536d259226507530cd26f45fa90"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.597244 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:54.617085 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:54.620605 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:54.620787 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:54.621423 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:26:54.622510 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:54.622568 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.622614 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:54.622637 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.634984 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38629
I20260925 16:26:54.635239 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38629 every 8 connection(s)
I20260925 16:26:54.636166 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:26:54.642650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 506
I20260925 16:26:54.642809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:26:54.643143 392 external_mini_cluster.cc:1428] Setting key 7e96e9d70057562e17505d75422aab18
Sep 25 16:26:54 dist-test-slave-31kt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:26:54.656958 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:54.639125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58069 (local address 127.0.98.62:37919)
0925 16:26:54.639430 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:54.639437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:54.640798 (+ 1361us) server_negotiation.cc:407] Connection header received
0925 16:26:54.642716 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:54.642722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:54.642812 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:54.642930 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:54.644872 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.645532 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.647967 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.648351 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.648510 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:54.652231 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:54.652236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:54.652239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:54.652275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:54.654171 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:54.654934 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:54.654938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:54.654939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:54.655033 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:54.655474 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:54.655478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:54.655480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:54.655717 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:54.655812 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:54.656542 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 16:26:54.656731 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 16:26:54.658210 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:54.639782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37919 (local address 127.0.98.1:58069)
0925 16:26:54.640555 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:54.640625 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:54.642318 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:54.643192 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:54.643209 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:54.643752 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:54.644639 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:54.644661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.645703 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.645708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:54.647664 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:54.647692 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.648615 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.648621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:54.648908 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:54.649856 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:54.649877 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:54.651939 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:54.654367 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:54.654377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:54.654401 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:54.654792 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:54.655123 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:54.655126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:54.655129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:54.655324 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:54.655837 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:54.655847 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:54.656239 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 16:26:54.656470 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 16:26:54.656956 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":207,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:26:54.660209 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37919
I20260925 16:26:54.660648 635 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:54.661460 635 heartbeater.cc:511] Master 127.0.98.62:37919 requested a full tablet report, sending...
I20260925 16:26:54.664074 438 ts_manager.cc:194] Registered new tserver with Master: badd2379120c484899492b5fed6700dd (127.0.98.1:38629)
I20260925 16:26:54.665723 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:26:54.678375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37919
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:54.813846 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:54.814296 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:54.814352 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:54.820158 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:54.820264 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:54.820344 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:26:54.826906 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:54.828555 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:54.830169 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:54.843362 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:54.843590 640 server_base.cc:1034] running on GCE node
W20260925 16:26:54.843426 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:54.843426 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:54.844381 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:54.845305 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:54.846567 640 hybrid_clock.cc:648] HybridClock initialized: now 1790353614846524 us; error 69 us; skew 500 ppm
Sep 25 16:26:54 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:54.850485 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:26:54.851850 640 webserver.cc:460] Webserver started at http://127.0.98.2:37321/ using document root <none> and password file <none>
I20260925 16:26:54.852612 640 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:54.852692 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:54.852936 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:54.855863 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ee83b0770794fc5bc02ba466e6b838b"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "fae02d82e1d86ba748f1c86f2ebc4ef6"
server_key_iv: "b4fdbad4a076a787c25f21b35e791e49"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.856630 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ee83b0770794fc5bc02ba466e6b838b"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "fae02d82e1d86ba748f1c86f2ebc4ef6"
server_key_iv: "b4fdbad4a076a787c25f21b35e791e49"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.862062 640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:26:54.865667 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.867630 640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:26:54.867826 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "9ee83b0770794fc5bc02ba466e6b838b"
format_stamp: "Formatted at 2026-09-25 16:26:54 on dist-test-slave-31kt"
server_key: "fae02d82e1d86ba748f1c86f2ebc4ef6"
server_key_iv: "b4fdbad4a076a787c25f21b35e791e49"
server_key_version: "encryptionkey@0"
I20260925 16:26:54.867998 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:54.890467 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:54.894199 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:54.894392 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:54.895045 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:26:54.896030 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:54.896083 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.896138 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:54.896166 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:54.907366 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40347
I20260925 16:26:54.907442 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40347 every 8 connection(s)
I20260925 16:26:54.908443 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:26:54.917558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 640
I20260925 16:26:54.917687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:26:54.918012 392 external_mini_cluster.cc:1428] Setting key d0ca07a8cbf2418d62dbe245049664dc
Sep 25 16:26:54 dist-test-slave-31kt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:54.926432 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:54.911015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47831 (local address 127.0.98.62:37919)
0925 16:26:54.911292 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:54.911297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:54.912594 (+ 1297us) server_negotiation.cc:407] Connection header received
0925 16:26:54.914232 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:54.914236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:54.914347 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:54.914466 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:54.916054 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.916702 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.918171 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.918485 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.918593 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:54.921661 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:54.921666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:54.921669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:54.921707 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:54.923471 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:54.924325 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:54.924331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:54.924332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:54.924480 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:54.924944 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:54.924946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:54.924947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:54.925163 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:54.925249 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:54.926142 (+ 893us) server_negotiation.cc:299] Negotiation successful
0925 16:26:54.926292 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260925 16:26:54.928006 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:54.911549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37919 (local address 127.0.98.2:47831)
0925 16:26:54.912330 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:54.912401 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:54.913915 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:54.914759 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:54.914769 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:54.915224 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:54.915818 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:54.915836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.916875 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.916879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:54.917996 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:54.918009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:54.918719 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:54.918723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:54.918949 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:54.919764 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:54.919787 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:54.921419 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:54.923645 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:54.923656 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:54.923680 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:54.924157 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:54.924616 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:54.924619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:54.924621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:54.924852 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:54.925269 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:54.925278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:54.925795 (+ 517us) client_negotiation.cc:769] Sending connection context
0925 16:26:54.926082 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 16:26:54.926565 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":192,"threads_started":1}
I20260925 16:26:54.929711 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37919
I20260925 16:26:54.930133 769 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:26:54.930800 769 heartbeater.cc:511] Master 127.0.98.62:37919 requested a full tablet report, sending...
I20260925 16:26:54.932399 438 ts_manager.cc:194] Registered new tserver with Master: 9ee83b0770794fc5bc02ba466e6b838b (127.0.98.2:40347)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:26:54.933513 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:26:54.954226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37919
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:55.076164 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:55.076490 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:55.076538 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:55.081679 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:55.081847 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:55.081969 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:26:55.090301 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:55.091605 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:55.092793 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:55.104928 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:55.104905 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:55.104950 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:55.105192 774 server_base.cc:1034] running on GCE node
I20260925 16:26:55.106204 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:55.107024 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:55.108304 774 hybrid_clock.cc:648] HybridClock initialized: now 1790353615108230 us; error 108 us; skew 500 ppm
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:55.111557 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:26:55.112584 774 webserver.cc:460] Webserver started at http://127.0.98.3:38853/ using document root <none> and password file <none>
I20260925 16:26:55.113209 774 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:55.113277 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:55.113458 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:55.116477 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "f886304ce73d4616808582fed70cfabe"
format_stamp: "Formatted at 2026-09-25 16:26:55 on dist-test-slave-31kt"
server_key: "1c2cf0de880ef457f4bdc8989d408ee0"
server_key_iv: "67a0af7fcf3e46b16f1ab8775cc2c3e2"
server_key_version: "encryptionkey@0"
I20260925 16:26:55.117240 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f886304ce73d4616808582fed70cfabe"
format_stamp: "Formatted at 2026-09-25 16:26:55 on dist-test-slave-31kt"
server_key: "1c2cf0de880ef457f4bdc8989d408ee0"
server_key_iv: "67a0af7fcf3e46b16f1ab8775cc2c3e2"
server_key_version: "encryptionkey@0"
I20260925 16:26:55.122653 774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:26:55.126436 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:55.128755 774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:26:55.129000 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "f886304ce73d4616808582fed70cfabe"
format_stamp: "Formatted at 2026-09-25 16:26:55 on dist-test-slave-31kt"
server_key: "1c2cf0de880ef457f4bdc8989d408ee0"
server_key_iv: "67a0af7fcf3e46b16f1ab8775cc2c3e2"
server_key_version: "encryptionkey@0"
I20260925 16:26:55.129225 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:55.147531 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:55.151801 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:55.152034 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:55.152803 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:26:55.153908 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:55.153964 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:55.154014 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:55.154030 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:55.168689 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41043
I20260925 16:26:55.168704 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41043 every 8 connection(s)
I20260925 16:26:55.170143 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:26:55.173905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 774
I20260925 16:26:55.174042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:26:55.174590 392 external_mini_cluster.cc:1428] Setting key 3606daf4a224de7dde97e2b2b76aa4ca
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:55.190127 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:55.172845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40227 (local address 127.0.98.62:37919)
0925 16:26:55.173052 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:55.173057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:55.174367 (+ 1310us) server_negotiation.cc:407] Connection header received
0925 16:26:55.176106 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:55.176110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:55.176186 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:55.176285 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:55.178242 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.178892 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.180552 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.180841 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.180965 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:55.184535 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:55.184539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:55.184542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:55.184580 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:55.186634 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:55.187529 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:55.187534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:55.187536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:55.187663 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:55.188300 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:55.188306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:55.188307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:55.188625 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:55.188742 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:55.189744 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0925 16:26:55.189959 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:26:55.191759 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:55.173384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37919 (local address 127.0.98.3:40227)
0925 16:26:55.174052 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:55.174122 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:55.175783 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:55.176578 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:55.176591 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:55.177208 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:55.178042 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:55.178072 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.179087 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.179090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:55.180319 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:55.180338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.181066 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.181069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:55.181285 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:55.182145 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:55.182167 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:55.184180 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:55.186854 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:55.186864 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:55.186887 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:55.187320 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:55.187862 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:55.187865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:55.187867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:55.188077 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:55.188809 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:55.188821 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:55.189363 (+ 542us) client_negotiation.cc:769] Sending connection context
0925 16:26:55.189636 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 16:26:55.190217 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":152,"threads_started":1}
I20260925 16:26:55.193392 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37919
I20260925 16:26:55.193738 903 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:55.194471 903 heartbeater.cc:511] Master 127.0.98.62:37919 requested a full tablet report, sending...
I20260925 16:26:55.195744 438 ts_manager.cc:194] Registered new tserver with Master: f886304ce73d4616808582fed70cfabe (127.0.98.3:41043)
I20260925 16:26:55.196354 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40227
I20260925 16:26:55.204842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:26:55.239980 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:55.226430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36762 (local address 127.0.98.1:38629)
0925 16:26:55.226860 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:55.226868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:55.226906 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:26:55.227062 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:55.227069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:55.227319 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:55.227467 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:55.228624 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.229472 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.230787 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.231268 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.231393 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:55.235275 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:55.235290 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:55.235308 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:55.235349 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:55.237186 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:55.237951 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:55.237966 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:55.237968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:55.238109 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:55.238588 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:55.238592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:55.238611 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:55.238969 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:55.239190 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:55.239553 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 16:26:55.239772 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
W20260925 16:26:55.241632 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36762
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:26:55.277596 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:55.265003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.0.98.1:38629)
0925 16:26:55.265176 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:55.265181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:55.265345 (+ 164us) server_negotiation.cc:407] Connection header received
0925 16:26:55.265506 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:55.265509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:55.265575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:55.265721 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:55.266807 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.267583 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.268738 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:55.269024 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:55.269180 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:55.272852 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:55.272858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:55.272860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:55.272909 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:55.274940 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:55.275725 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:55.275731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:55.275733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:55.275937 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:55.276314 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:55.276318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:55.276320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:55.276571 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:55.276738 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:55.277232 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 16:26:55.277410 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:26:55.279819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 506
I20260925 16:26:55.286270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 640
I20260925 16:26:55.296072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 774
I20260925 16:26:55.302690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 412
2026-09-25T16:26:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2859 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[917](info): set up 2 sockets
Sep 25 16:26:55 dist-test-slave-31kt krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 16:26:56.301815 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:26:56 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:26:56.443861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:56.583200 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:56.583537 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:56.583588 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:56.588378 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:26:56.588466 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:56.588485 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:56.588501 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:26:56.588517 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:26:56.595094 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:56.596366 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:56.597534 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:56.609297 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:56.609315 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:56.610034 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:56.610136 930 server_base.cc:1034] running on GCE node
I20260925 16:26:56.611028 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:56.612974 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:56.614244 930 hybrid_clock.cc:648] HybridClock initialized: now 1790353616614212 us; error 73 us; skew 500 ppm
Sep 25 16:26:56 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:56.617722 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:26:56.619045 930 webserver.cc:460] Webserver started at http://127.0.98.62:42071/ using document root <none> and password file <none>
I20260925 16:26:56.619782 930 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:56.619860 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:56.620095 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:56.622987 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "ec6b9068095841e4891dfa2de0521d5f"
format_stamp: "Formatted at 2026-09-25 16:26:56 on dist-test-slave-31kt"
server_key: "2f2b4508a1af948ab777be0fff5891f9"
server_key_iv: "e65b67fa2c2d824e92b9c3d0b67a81c8"
server_key_version: "encryptionkey@0"
I20260925 16:26:56.623656 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec6b9068095841e4891dfa2de0521d5f"
format_stamp: "Formatted at 2026-09-25 16:26:56 on dist-test-slave-31kt"
server_key: "2f2b4508a1af948ab777be0fff5891f9"
server_key_iv: "e65b67fa2c2d824e92b9c3d0b67a81c8"
server_key_version: "encryptionkey@0"
I20260925 16:26:56.629137 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:26:56.632738 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:56.634809 930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:26:56.634999 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "ec6b9068095841e4891dfa2de0521d5f"
format_stamp: "Formatted at 2026-09-25 16:26:56 on dist-test-slave-31kt"
server_key: "2f2b4508a1af948ab777be0fff5891f9"
server_key_iv: "e65b67fa2c2d824e92b9c3d0b67a81c8"
server_key_version: "encryptionkey@0"
I20260925 16:26:56.635162 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:56.661082 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:56.665568 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:56.665877 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:56.674976 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32825
I20260925 16:26:56.675024 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32825 every 8 connection(s)
I20260925 16:26:56.676282 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:26:56.680212 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:26:56.683930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 930
I20260925 16:26:56.684077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:26:56.684386 392 external_mini_cluster.cc:1428] Setting key 05016f228b85bea09d5d9425d572bbd3
I20260925 16:26:56.687976 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f: Bootstrap starting.
I20260925 16:26:56.691411 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f: Neither blocks nor log segments found. Creating new log.
I20260925 16:26:56.692442 998 log.cc:824] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:26:56.694971 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f: No bootstrap required, opened a new log
Sep 25 16:26:56 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:56.698515 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6b9068095841e4891dfa2de0521d5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32825 } }
I20260925 16:26:56.698828 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:26:56.698897 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6b9068095841e4891dfa2de0521d5f, State: Initialized, Role: FOLLOWER
I20260925 16:26:56.699471 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6b9068095841e4891dfa2de0521d5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32825 } }
I20260925 16:26:56.699661 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:26:56.699728 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:26:56.699841 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:26:56.700971 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6b9068095841e4891dfa2de0521d5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32825 } }
I20260925 16:26:56.701395 998 leader_election.cc:302] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec6b9068095841e4891dfa2de0521d5f; no voters:
I20260925 16:26:56.702109 998 leader_election.cc:288] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:26:56.702210 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:26:56.702440 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [term 1 LEADER]: Becoming Leader. State: Replica: ec6b9068095841e4891dfa2de0521d5f, State: Running, Role: LEADER
I20260925 16:26:56.702804 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6b9068095841e4891dfa2de0521d5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32825 } }
I20260925 16:26:56.703246 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:56.686198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.98.62:32825)
0925 16:26:56.686905 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:56.686933 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:56.686992 (+ 59us) server_negotiation.cc:407] Connection header received
0925 16:26:56.688158 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:56.688198 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:56.688806 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:56.689309 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:56.690378 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:56.691832 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:56.692849 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:56.693451 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:56.693624 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:56.696410 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:56.696417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:56.696439 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:56.696478 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:56.698949 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:56.699778 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:56.699786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:56.699795 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:56.699993 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:56.700351 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:56.700354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:56.700355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:56.700930 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:56.701144 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:56.701486 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:26:56.701921 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":214,"threads_started":1}
I20260925 16:26:56.703713 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:26:56.704845 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec6b9068095841e4891dfa2de0521d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6b9068095841e4891dfa2de0521d5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32825 } } }
I20260925 16:26:56.705039 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [sys.catalog]: This master's current role is: LEADER
I20260925 16:26:56.705391 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec6b9068095841e4891dfa2de0521d5f. Latest consensus state: current_term: 1 leader_uuid: "ec6b9068095841e4891dfa2de0521d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6b9068095841e4891dfa2de0521d5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32825 } } }
I20260925 16:26:56.705507 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f [sys.catalog]: This master's current role is: LEADER
W20260925 16:26:56.709388 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:26:56.709491 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:26:56.709556 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:26:56.710368 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:26:56.717828 1012 catalog_manager.cc:1347] Generated new cluster ID: 3d8474ba34054eaeb06890fdcad6fcb9
I20260925 16:26:56.717943 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:26:56.749940 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:26:56.751073 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:26:56.762496 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec6b9068095841e4891dfa2de0521d5f: Generated new TSK 0
I20260925 16:26:56.763405 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:26:56.806900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32825
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:56.944677 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:56.945016 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:56.945080 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:56.950301 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:56.950409 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:56.950505 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:26:56.956549 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:56.957783 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:56.959113 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:56.970638 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:56.970767 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:56.970667 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:56.971032 1024 server_base.cc:1034] running on GCE node
I20260925 16:26:56.971719 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:56.972472 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:56.973687 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790353616973645 us; error 67 us; skew 500 ppm
Sep 25 16:26:56 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:56.977135 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:26:56.978366 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46825/ using document root <none> and password file <none>
I20260925 16:26:56.979048 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:56.979128 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:56.979580 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:56.982296 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ef6dcdbfc99142189a2a2738b00cc9dc"
format_stamp: "Formatted at 2026-09-25 16:26:56 on dist-test-slave-31kt"
server_key: "5c17f89967a0842f6c9074163a1b105d"
server_key_iv: "c73dd4d98cc9d022745277a581b820d9"
server_key_version: "encryptionkey@0"
I20260925 16:26:56.982968 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ef6dcdbfc99142189a2a2738b00cc9dc"
format_stamp: "Formatted at 2026-09-25 16:26:56 on dist-test-slave-31kt"
server_key: "5c17f89967a0842f6c9074163a1b105d"
server_key_iv: "c73dd4d98cc9d022745277a581b820d9"
server_key_version: "encryptionkey@0"
I20260925 16:26:56.987941 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:26:56.991256 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:56.992954 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:26:56.993142 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "ef6dcdbfc99142189a2a2738b00cc9dc"
format_stamp: "Formatted at 2026-09-25 16:26:56 on dist-test-slave-31kt"
server_key: "5c17f89967a0842f6c9074163a1b105d"
server_key_iv: "c73dd4d98cc9d022745277a581b820d9"
server_key_version: "encryptionkey@0"
I20260925 16:26:56.993275 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:57.020861 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:57.024787 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:57.025032 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:57.025718 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:26:57.026989 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:57.027055 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.027129 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:57.027158 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.038108 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38981
I20260925 16:26:57.038224 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38981 every 8 connection(s)
I20260925 16:26:57.039244 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:26:57.046155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1024
I20260925 16:26:57.046308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:26:57.046834 392 external_mini_cluster.cc:1428] Setting key 763dd2b34d8aae0546ba5e3c10313a77
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:26:57.058346 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.042012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38171 (local address 127.0.98.62:32825)
0925 16:26:57.042212 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.042218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.043373 (+ 1155us) server_negotiation.cc:407] Connection header received
0925 16:26:57.045203 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.045207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.045290 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.045380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.047516 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.048292 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.049919 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.050301 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.050481 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.053954 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.053959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.053962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.053997 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.055763 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.056426 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.056430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.056431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.056515 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.056877 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.056880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.056881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.057115 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.057205 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.057941 (+ 736us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.058130 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:26:57.059538 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.042484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32825 (local address 127.0.98.1:38171)
0925 16:26:57.043140 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.043218 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.044849 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.045673 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.045684 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.046379 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:57.047227 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.047248 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.048479 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.048483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.049669 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.049684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.050585 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.050589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.050864 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.051764 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:57.051789 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:57.053624 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:57.055927 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:57.055935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:57.055961 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:57.056304 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:57.056604 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:57.056607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:57.056610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:57.056778 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:57.057228 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:57.057237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:57.057624 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.057876 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.058358 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:26:57.061378 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32825
I20260925 16:26:57.061697 1153 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:57.062460 1153 heartbeater.cc:511] Master 127.0.98.62:32825 requested a full tablet report, sending...
I20260925 16:26:57.064666 962 ts_manager.cc:194] Registered new tserver with Master: ef6dcdbfc99142189a2a2738b00cc9dc (127.0.98.1:38981)
I20260925 16:26:57.066296 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:26:57.082602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32825
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:57.215428 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:57.215857 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:57.215914 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:57.220799 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:57.220902 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:57.220978 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:26:57.227499 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:57.229075 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:57.230511 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:57.243659 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:57.243659 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:57.243837 1158 server_base.cc:1034] running on GCE node
W20260925 16:26:57.243677 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:57.244730 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:57.245790 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:57.247087 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790353617247036 us; error 70 us; skew 500 ppm
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:57.250993 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:26:57.252264 1158 webserver.cc:460] Webserver started at http://127.0.98.2:33809/ using document root <none> and password file <none>
I20260925 16:26:57.252942 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:57.253023 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:57.253216 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:57.255744 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "451c78db8cac4732acbe25f7ac9ea5c4"
format_stamp: "Formatted at 2026-09-25 16:26:57 on dist-test-slave-31kt"
server_key: "7adf4fe06ed03be09cd113046549ea61"
server_key_iv: "9ce676b487b9461b818e1f835b8eb673"
server_key_version: "encryptionkey@0"
I20260925 16:26:57.256381 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "451c78db8cac4732acbe25f7ac9ea5c4"
format_stamp: "Formatted at 2026-09-25 16:26:57 on dist-test-slave-31kt"
server_key: "7adf4fe06ed03be09cd113046549ea61"
server_key_iv: "9ce676b487b9461b818e1f835b8eb673"
server_key_version: "encryptionkey@0"
I20260925 16:26:57.261336 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:26:57.264782 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.266595 1158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:26:57.266762 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "451c78db8cac4732acbe25f7ac9ea5c4"
format_stamp: "Formatted at 2026-09-25 16:26:57 on dist-test-slave-31kt"
server_key: "7adf4fe06ed03be09cd113046549ea61"
server_key_iv: "9ce676b487b9461b818e1f835b8eb673"
server_key_version: "encryptionkey@0"
I20260925 16:26:57.266887 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:57.282615 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:57.286803 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:57.286998 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:57.287662 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:26:57.288659 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:57.288718 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.288765 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:57.288779 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.299141 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40827
I20260925 16:26:57.299192 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40827 every 8 connection(s)
I20260925 16:26:57.300202 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:26:57.301561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1158
I20260925 16:26:57.301712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:26:57.302075 392 external_mini_cluster.cc:1428] Setting key 50f565ca44fa11cab6fb392e4f63c04b
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:57.323681 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.303081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55835 (local address 127.0.98.62:32825)
0925 16:26:57.303306 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.303311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.305037 (+ 1726us) server_negotiation.cc:407] Connection header received
0925 16:26:57.307256 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.307262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.307343 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.307454 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.309277 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.310054 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.312307 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.312582 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.312695 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.318602 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.318608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.318612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.318663 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.320278 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.321112 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.321116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.321118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.321243 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.321830 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.321835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.321836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.322121 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.322230 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.323311 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.323500 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:26:57.325212 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.303777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32825 (local address 127.0.98.2:55835)
0925 16:26:57.304774 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.304863 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.306930 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.307721 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.307732 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.308244 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:57.308945 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.308972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.310267 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.310272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.312066 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.312085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.312795 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.312801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.313069 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.313908 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:57.313933 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:57.318275 (+ 4342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:57.320452 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:57.320460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:57.320487 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:57.320960 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:57.321352 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:57.321355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:57.321358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:57.321605 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:57.322248 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:57.322257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:57.322753 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.323079 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.323619 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":342,"threads_started":1}
I20260925 16:26:57.327162 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32825
I20260925 16:26:57.327601 1287 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:57.328306 1287 heartbeater.cc:511] Master 127.0.98.62:32825 requested a full tablet report, sending...
I20260925 16:26:57.330029 962 ts_manager.cc:194] Registered new tserver with Master: 451c78db8cac4732acbe25f7ac9ea5c4 (127.0.98.2:40827)
I20260925 16:26:57.330706 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:26:57.337316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32825
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:57.479915 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:57.480382 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:57.480455 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:57.485443 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:57.485551 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:57.485656 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:26:57.493539 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:57.495360 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:57.496856 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:57.510403 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:57.510435 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:57.510522 1292 server_base.cc:1034] running on GCE node
W20260925 16:26:57.510435 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:57.511308 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:57.512181 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:57.513510 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790353617513485 us; error 52 us; skew 500 ppm
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:57.521045 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:26:57.522238 1292 webserver.cc:460] Webserver started at http://127.0.98.3:32883/ using document root <none> and password file <none>
I20260925 16:26:57.522902 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:57.522972 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:57.523187 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:57.525609 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "544fd6d8afec4f7e9db4d6e04ee44d92"
format_stamp: "Formatted at 2026-09-25 16:26:57 on dist-test-slave-31kt"
server_key: "b0999c3e372f66752c2551c039b7338b"
server_key_iv: "092a6a2acf4eabf277faf66eea8ec9fa"
server_key_version: "encryptionkey@0"
I20260925 16:26:57.526309 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "544fd6d8afec4f7e9db4d6e04ee44d92"
format_stamp: "Formatted at 2026-09-25 16:26:57 on dist-test-slave-31kt"
server_key: "b0999c3e372f66752c2551c039b7338b"
server_key_iv: "092a6a2acf4eabf277faf66eea8ec9fa"
server_key_version: "encryptionkey@0"
I20260925 16:26:57.531144 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:26:57.534703 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.536653 1292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:26:57.536845 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "544fd6d8afec4f7e9db4d6e04ee44d92"
format_stamp: "Formatted at 2026-09-25 16:26:57 on dist-test-slave-31kt"
server_key: "b0999c3e372f66752c2551c039b7338b"
server_key_iv: "092a6a2acf4eabf277faf66eea8ec9fa"
server_key_version: "encryptionkey@0"
I20260925 16:26:57.536993 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:57.559315 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:57.563159 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:57.563365 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:57.564035 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:26:57.564988 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:57.565044 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.565099 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:57.565126 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:57.576923 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46363
I20260925 16:26:57.577045 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46363 every 8 connection(s)
I20260925 16:26:57.578104 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:26:57.587124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1292
I20260925 16:26:57.587337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:26:57.587678 392 external_mini_cluster.cc:1428] Setting key 9ab3b6141d054c5f060f7bea139d19a1
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:57.596350 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.580735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60267 (local address 127.0.98.62:32825)
0925 16:26:57.580980 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.580986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.582187 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 16:26:57.583903 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.583907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.583977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.584091 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.585724 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.586384 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.588203 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.588555 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.588664 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.591834 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.591838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.591841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.591879 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.593518 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.594342 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.594345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.594346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.594427 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.594908 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.594911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.594912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.595190 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.595280 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.595994 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.596155 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 16:26:57.597637 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.581219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32825 (local address 127.0.98.3:60267)
0925 16:26:57.581953 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.582029 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.583561 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.584333 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.584342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.584862 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:57.585515 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.585531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.586542 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.586545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.588023 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.588036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.588806 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.588811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.589015 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.589900 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:57.589928 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:57.591602 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:57.593745 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:57.593808 (+ 63us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:57.593837 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:57.594238 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:57.594540 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:57.594543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:57.594545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:57.594750 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:57.595310 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:57.595317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:57.595689 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.595902 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.596379 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":156,"threads_started":1}
I20260925 16:26:57.599328 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32825
I20260925 16:26:57.599668 1421 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:57.600390 1421 heartbeater.cc:511] Master 127.0.98.62:32825 requested a full tablet report, sending...
I20260925 16:26:57.602310 961 ts_manager.cc:194] Registered new tserver with Master: 544fd6d8afec4f7e9db4d6e04ee44d92 (127.0.98.3:46363)
I20260925 16:26:57.603188 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60267
I20260925 16:26:57.603385 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:26:57.618274 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.608480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.98.62:32825)
0925 16:26:57.608674 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.608679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.609285 (+ 606us) server_negotiation.cc:407] Connection header received
0925 16:26:57.609394 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.609397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.609452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.609553 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.610491 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.611049 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.612033 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.612301 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.612424 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.614228 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.614233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.614236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.614274 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.615821 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.616392 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.616395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.616397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.616500 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.616916 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.616920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.616922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.617300 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.617439 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.617801 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.618073 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:26:57.625114 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:26:57.628645 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:26:57.648129 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.640897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.0.98.3:46363)
0925 16:26:57.641353 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.641367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.642149 (+ 782us) server_negotiation.cc:407] Connection header received
0925 16:26:57.642561 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.642569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.642857 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.643035 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:26:57.643356 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.644544 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.646477 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.647277 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.647391 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.647756 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.647828 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.647951 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":169,"threads_started":1}
I20260925 16:26:57.648165 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.640787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46363 (local address 127.0.0.1:43910)
0925 16:26:57.641937 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.642003 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.642200 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.643031 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.643038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.643061 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:26:57.643244 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.643251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.644713 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.644716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.646316 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.646329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.647407 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.647411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.647587 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.647642 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.647768 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.647945 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":34,"thread_start_us":150,"threads_started":1}
I20260925 16:26:57.649011 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.642289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38981 (local address 127.0.0.1:51036)
0925 16:26:57.642678 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.642693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.642793 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.643614 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.643617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.643636 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:26:57.643783 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.643815 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.645213 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.645216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.646157 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.646167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.647404 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.647409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.648606 (+ 1197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.648642 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.648726 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.648827 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":23,"thread_start_us":100,"threads_started":1}
I20260925 16:26:57.649178 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.640829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40827 (local address 127.0.0.1:47548)
0925 16:26:57.642660 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.642675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.642771 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.643456 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.643459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.643476 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:26:57.643570 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.643576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.644956 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.644960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.646332 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.646339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.648817 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.648821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.648913 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.648938 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.648981 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.649060 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"mutex_wait_us":88,"thread_start_us":138,"threads_started":1}
I20260925 16:26:57.649192 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.642514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.0.98.1:38981)
0925 16:26:57.643003 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.643011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.643031 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:26:57.643100 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.643104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.643258 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.643440 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:26:57.643978 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.645077 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.646312 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.647236 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.647373 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.647557 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.648809 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.648976 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":139,"threads_started":1}
I20260925 16:26:57.650046 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.641952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.98.2:40827)
0925 16:26:57.642411 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.642424 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.642867 (+ 443us) server_negotiation.cc:407] Connection header received
0925 16:26:57.642973 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.642980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.643259 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.643446 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:26:57.643703 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.644789 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.647531 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.648672 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.649499 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.649693 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.649746 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.649883 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":138,"threads_started":1}
I20260925 16:26:57.652522 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 857041d3edd24c19b8070f8d3633ae4f (DEFAULT_TABLE table=test-table [id=8f5d6df2eb784d85aaeffd95f0c6be94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:26:57.653163 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 857041d3edd24c19b8070f8d3633ae4f (DEFAULT_TABLE table=test-table [id=8f5d6df2eb784d85aaeffd95f0c6be94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:26:57.652904 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 857041d3edd24c19b8070f8d3633ae4f (DEFAULT_TABLE table=test-table [id=8f5d6df2eb784d85aaeffd95f0c6be94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:26:57.653945 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857041d3edd24c19b8070f8d3633ae4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:26:57.654438 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857041d3edd24c19b8070f8d3633ae4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:26:57.654438 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857041d3edd24c19b8070f8d3633ae4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:26:57.662798 1438 tablet_bootstrap.cc:492] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: Bootstrap starting.
I20260925 16:26:57.664980 1439 tablet_bootstrap.cc:492] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: Bootstrap starting.
I20260925 16:26:57.666072 1440 tablet_bootstrap.cc:492] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: Bootstrap starting.
I20260925 16:26:57.668303 1438 tablet_bootstrap.cc:654] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: Neither blocks nor log segments found. Creating new log.
I20260925 16:26:57.668303 1439 tablet_bootstrap.cc:654] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: Neither blocks nor log segments found. Creating new log.
I20260925 16:26:57.668916 1440 tablet_bootstrap.cc:654] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: Neither blocks nor log segments found. Creating new log.
I20260925 16:26:57.669600 1439 log.cc:824] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: Log is configured to *not* fsync() on all Append() calls
I20260925 16:26:57.670380 1438 log.cc:824] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: Log is configured to *not* fsync() on all Append() calls
I20260925 16:26:57.671046 1440 log.cc:824] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:26:57.673086 1439 tablet_bootstrap.cc:492] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: No bootstrap required, opened a new log
I20260925 16:26:57.673427 1439 ts_tablet_manager.cc:1397] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 16:26:57.674633 1438 tablet_bootstrap.cc:492] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: No bootstrap required, opened a new log
I20260925 16:26:57.674623 1440 tablet_bootstrap.cc:492] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: No bootstrap required, opened a new log
I20260925 16:26:57.674902 1438 ts_tablet_manager.cc:1397] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20260925 16:26:57.674902 1440 ts_tablet_manager.cc:1397] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 16:26:57.680034 1438 raft_consensus.cc:348] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.680034 1440 raft_consensus.cc:348] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.680470 1438 raft_consensus.cc:374] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:26:57.680470 1440 raft_consensus.cc:374] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:26:57.680542 1440 raft_consensus.cc:729] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c78db8cac4732acbe25f7ac9ea5c4, State: Initialized, Role: FOLLOWER
I20260925 16:26:57.680540 1438 raft_consensus.cc:729] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544fd6d8afec4f7e9db4d6e04ee44d92, State: Initialized, Role: FOLLOWER
I20260925 16:26:57.681228 1440 consensus_queue.cc:260] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.681228 1438 consensus_queue.cc:260] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.682587 1440 ts_tablet_manager.cc:1428] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260925 16:26:57.682587 1438 ts_tablet_manager.cc:1428] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260925 16:26:57.683055 1287 heartbeater.cc:503] Master 127.0.98.62:32825 was elected leader, sending a full tablet report...
I20260925 16:26:57.683133 1421 heartbeater.cc:503] Master 127.0.98.62:32825 was elected leader, sending a full tablet report...
I20260925 16:26:57.685077 1439 raft_consensus.cc:348] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.685613 1439 raft_consensus.cc:374] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:26:57.685724 1439 raft_consensus.cc:729] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6dcdbfc99142189a2a2738b00cc9dc, State: Initialized, Role: FOLLOWER
I20260925 16:26:57.686508 1439 consensus_queue.cc:260] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.688068 1439 ts_tablet_manager.cc:1428] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260925 16:26:57.688512 1153 heartbeater.cc:503] Master 127.0.98.62:32825 was elected leader, sending a full tablet report...
I20260925 16:26:57.709589 1444 raft_consensus.cc:482] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:26:57.710026 1444 raft_consensus.cc:504] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.711812 1444 leader_election.cc:288] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451c78db8cac4732acbe25f7ac9ea5c4 (127.0.98.2:40827), ef6dcdbfc99142189a2a2738b00cc9dc (127.0.98.1:38981)
I20260925 16:26:57.717175 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.712253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38981 (local address 127.0.98.3:41283)
0925 16:26:57.712530 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.712550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.712705 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.713087 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.713091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.713116 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:26:57.713285 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.713293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.714315 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.714320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.715355 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.715365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.716641 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.716645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.716771 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.716797 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.716885 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.716981 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260925 16:26:57.717212 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.712415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41283 (local address 127.0.98.1:38981)
0925 16:26:57.712596 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.712601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.712629 (+ 28us) server_negotiation.cc:407] Connection header received
0925 16:26:57.712866 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.712870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.712938 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.713027 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:26:57.713437 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.714177 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.715542 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.716396 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.716541 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.716614 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.716932 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.717042 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 16:26:57.717411 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.712318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40885 (local address 127.0.98.2:40827)
0925 16:26:57.712509 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.712515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.712969 (+ 454us) server_negotiation.cc:407] Connection header received
0925 16:26:57.713089 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.713092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.713157 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.713319 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:26:57.713941 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.714670 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.716086 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.716622 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.716754 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.716838 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.717176 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.717282 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 16:26:57.717923 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.712253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40827 (local address 127.0.98.3:40885)
0925 16:26:57.712749 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:57.712766 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:57.712968 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:57.713381 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:57.713404 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:57.713433 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:26:57.713600 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.713608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.714834 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.714837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.715925 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:57.715935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.716820 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.716825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:57.716976 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.717010 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:26:57.717116 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:26:57.717220 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":42,"thread_start_us":172,"threads_started":1}
I20260925 16:26:57.718279 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857041d3edd24c19b8070f8d3633ae4f" candidate_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" is_pre_election: true
I20260925 16:26:57.718704 1107 raft_consensus.cc:2454] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544fd6d8afec4f7e9db4d6e04ee44d92 in term 0.
I20260925 16:26:57.718915 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857041d3edd24c19b8070f8d3633ae4f" candidate_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" is_pre_election: true
I20260925 16:26:57.719435 1241 raft_consensus.cc:2454] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544fd6d8afec4f7e9db4d6e04ee44d92 in term 0.
I20260925 16:26:57.719465 1309 leader_election.cc:302] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544fd6d8afec4f7e9db4d6e04ee44d92, ef6dcdbfc99142189a2a2738b00cc9dc; no voters:
I20260925 16:26:57.719870 1444 raft_consensus.cc:2781] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:26:57.720024 1444 raft_consensus.cc:482] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:26:57.720077 1444 raft_consensus.cc:3037] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:26:57.721554 1444 raft_consensus.cc:504] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.722143 1444 leader_election.cc:288] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [CANDIDATE]: Term 1 election: Requested vote from peers 451c78db8cac4732acbe25f7ac9ea5c4 (127.0.98.2:40827), ef6dcdbfc99142189a2a2738b00cc9dc (127.0.98.1:38981)
I20260925 16:26:57.722792 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857041d3edd24c19b8070f8d3633ae4f" candidate_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc"
I20260925 16:26:57.722975 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857041d3edd24c19b8070f8d3633ae4f" candidate_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451c78db8cac4732acbe25f7ac9ea5c4"
I20260925 16:26:57.723011 1107 raft_consensus.cc:3037] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:26:57.723163 1241 raft_consensus.cc:3037] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:26:57.724670 1241 raft_consensus.cc:2454] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544fd6d8afec4f7e9db4d6e04ee44d92 in term 1.
I20260925 16:26:57.725188 1107 raft_consensus.cc:2454] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544fd6d8afec4f7e9db4d6e04ee44d92 in term 1.
I20260925 16:26:57.725258 1311 leader_election.cc:302] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451c78db8cac4732acbe25f7ac9ea5c4, 544fd6d8afec4f7e9db4d6e04ee44d92; no voters:
I20260925 16:26:57.725519 1444 raft_consensus.cc:2781] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:26:57.725906 1444 raft_consensus.cc:686] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [term 1 LEADER]: Becoming Leader. State: Replica: 544fd6d8afec4f7e9db4d6e04ee44d92, State: Running, Role: LEADER
I20260925 16:26:57.726411 1444 consensus_queue.cc:237] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } }
I20260925 16:26:57.730888 961 catalog_manager.cc:5495] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 544fd6d8afec4f7e9db4d6e04ee44d92 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544fd6d8afec4f7e9db4d6e04ee44d92" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451c78db8cac4732acbe25f7ac9ea5c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6dcdbfc99142189a2a2738b00cc9dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38981 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:26:57.755990 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.744564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.98.1:38981)
0925 16:26:57.744838 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.744844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.744991 (+ 147us) server_negotiation.cc:407] Connection header received
0925 16:26:57.745209 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.745212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.745272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.745382 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.746262 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.746852 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.747882 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.748126 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.748255 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.751421 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.751436 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.751457 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.751495 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.753600 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.754421 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.754428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.754429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.754530 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.754886 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.754888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.754889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.755213 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.755407 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.755668 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.755801 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:26:57.786845 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.776489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.98.1:38981)
0925 16:26:57.776708 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.776712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.776861 (+ 149us) server_negotiation.cc:407] Connection header received
0925 16:26:57.777007 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.777010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.777068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.777170 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.777985 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.778614 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.779522 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.779798 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.779890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.782792 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.782796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.782798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.782835 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.784501 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.785284 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.785287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.785288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.785385 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.785722 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.785725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.785726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.786025 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.786144 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.786497 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.786657 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 16:26:57.790900 1154 tablet.cc:2369] T 857041d3edd24c19b8070f8d3633ae4f P ef6dcdbfc99142189a2a2738b00cc9dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260925 16:26:57.801543 1288 tablet.cc:2369] T 857041d3edd24c19b8070f8d3633ae4f P 451c78db8cac4732acbe25f7ac9ea5c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:26:57.819217 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:57.808034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.0.98.1:38981)
0925 16:26:57.808253 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:57.808258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:57.808861 (+ 603us) server_negotiation.cc:407] Connection header received
0925 16:26:57.809063 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:57.809066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:57.809173 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:57.809286 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:57.810255 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.810827 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.811814 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:57.812060 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:57.812212 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:57.815229 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:57.815234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:57.815236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:57.815278 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:57.816907 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.817504 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.817508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.817509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.817630 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:57.818067 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:57.818070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:57.818072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:57.818299 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:57.818416 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:57.818812 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 16:26:57.818953 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260925 16:26:57.821146 1067 tablet_service.cc:2908] Rejecting scan request for tablet 857041d3edd24c19b8070f8d3633ae4f: Uninitialized: clean time has not yet been initialized
I20260925 16:26:57.824450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1024
W20260925 16:26:57.829356 1422 tablet.cc:2369] T 857041d3edd24c19b8070f8d3633ae4f P 544fd6d8afec4f7e9db4d6e04ee44d92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:26:57.831497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1158
I20260925 16:26:57.837693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1292
I20260925 16:26:57.844966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 930
2026-09-25T16:26:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2543 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[1462](info): set up 2 sockets
Sep 25 16:26:57 dist-test-slave-31kt krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260925 16:26:58.825845 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:26:58 dist-test-slave-31kt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:26:58.969784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:59.124521 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:59.124915 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:59.124977 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:59.129594 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:26:59.129695 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:26:59.129721 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:59.129746 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:59.129802 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:26:59.129824 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:26:59.136495 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37543
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:59.138177 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:59.139525 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:59.150895 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:59.150936 1475 server_base.cc:1034] running on GCE node
W20260925 16:26:59.150838 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:59.150867 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:59.152253 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:59.154023 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:59.155303 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790353619155249 us; error 88 us; skew 500 ppm
Sep 25 16:26:59 dist-test-slave-31kt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:59.159801 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:26:59.161164 1475 webserver.cc:460] Webserver started at http://127.0.98.62:43255/ using document root <none> and password file <none>
I20260925 16:26:59.162097 1475 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:59.162235 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:59.162539 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:59.165711 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "e2505cb233244cd995d19605add56638"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "162758b0d10a3828259cefa5b2450489"
server_key_iv: "a04743ed279bb06189249458a9eba367"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.166538 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2505cb233244cd995d19605add56638"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "162758b0d10a3828259cefa5b2450489"
server_key_iv: "a04743ed279bb06189249458a9eba367"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.172951 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 16:26:59.178592 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:59.181048 1475 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:26:59.181295 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "e2505cb233244cd995d19605add56638"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "162758b0d10a3828259cefa5b2450489"
server_key_iv: "a04743ed279bb06189249458a9eba367"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.181459 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:59.207227 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:59.211681 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:59.211949 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:59.223047 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37543
I20260925 16:26:59.223151 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37543 every 8 connection(s)
I20260925 16:26:59.224653 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:26:59.229517 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:26:59.230101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1475
I20260925 16:26:59.230243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:26:59.230621 392 external_mini_cluster.cc:1428] Setting key 3c0d729afb2012020fb6c58f986f2ea3
I20260925 16:26:59.237538 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: Bootstrap starting.
I20260925 16:26:59.240463 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: Neither blocks nor log segments found. Creating new log.
I20260925 16:26:59.241380 1553 log.cc:824] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:26:59 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:59.245193 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: No bootstrap required, opened a new log
I20260925 16:26:59.250272 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2505cb233244cd995d19605add56638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260925 16:26:59.250713 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:26:59.250819 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2505cb233244cd995d19605add56638, State: Initialized, Role: FOLLOWER
I20260925 16:26:59.251475 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2505cb233244cd995d19605add56638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260925 16:26:59.251696 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:26:59.251775 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:26:59.251885 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:26:59.252130 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.232602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.0.98.62:37543)
0925 16:26:59.233280 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:59.233302 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:59.233354 (+ 52us) server_negotiation.cc:407] Connection header received
0925 16:26:59.234381 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:59.234409 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:59.234969 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:59.235401 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:59.236329 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.237702 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.238752 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.239363 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.239547 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.242805 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:59.242815 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:59.242844 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:59.242882 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:59.246427 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.247303 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.247311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.247320 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.247523 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.248016 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.248019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.248021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.249407 (+ 1386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:59.249688 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:59.250272 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 16:26:59.250804 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":179,"threads_started":1}
I20260925 16:26:59.284248 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2505cb233244cd995d19605add56638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260925 16:26:59.284927 1553 leader_election.cc:302] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2505cb233244cd995d19605add56638; no voters:
I20260925 16:26:59.285481 1553 leader_election.cc:288] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:26:59.287026 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:26:59.287354 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [term 1 LEADER]: Becoming Leader. State: Replica: e2505cb233244cd995d19605add56638, State: Running, Role: LEADER
I20260925 16:26:59.287472 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:26:59.288100 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2505cb233244cd995d19605add56638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260925 16:26:59.290621 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2505cb233244cd995d19605add56638. Latest consensus state: current_term: 1 leader_uuid: "e2505cb233244cd995d19605add56638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2505cb233244cd995d19605add56638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } } }
I20260925 16:26:59.290822 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [sys.catalog]: This master's current role is: LEADER
I20260925 16:26:59.290702 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2505cb233244cd995d19605add56638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2505cb233244cd995d19605add56638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } } }
I20260925 16:26:59.290921 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638 [sys.catalog]: This master's current role is: LEADER
W20260925 16:26:59.294824 1572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:26:59.294946 1572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:26:59.295066 1573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:26:59.295996 1573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:26:59.304462 1573 catalog_manager.cc:1347] Generated new cluster ID: 1bcc5e89190c451aa8368e2da01c2a76
I20260925 16:26:59.304605 1573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:26:59.316923 1573 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:26:59.318624 1573 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:26:59.334712 1573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: Generated new TSK 0
I20260925 16:26:59.335798 1573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:26:59.442766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:59.579260 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:59.579591 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:59.579639 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:59.584005 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:59.584101 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:59.584143 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:26:59.584178 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:26:59.590404 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:59.591684 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:59.592844 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:59.604803 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:59.604803 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:59.604803 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:59.605268 1579 server_base.cc:1034] running on GCE node
I20260925 16:26:59.605967 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:59.606963 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:59.610361 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790353619610265 us; error 96 us; skew 500 ppm
Sep 25 16:26:59 dist-test-slave-31kt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:59.615348 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:26:59.617113 1579 webserver.cc:460] Webserver started at http://127.0.98.1:42383/ using document root <none> and password file <none>
I20260925 16:26:59.617882 1579 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:59.617967 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:59.618208 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:59.620710 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "bdf8009b1c954199a2f3e88ff33ebf9f"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "e9e27a3bd6a1a09cf0b15afc43852bb5"
server_key_iv: "47c2de6673ff96bb3654eb446a12acdc"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.621868 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bdf8009b1c954199a2f3e88ff33ebf9f"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "e9e27a3bd6a1a09cf0b15afc43852bb5"
server_key_iv: "47c2de6673ff96bb3654eb446a12acdc"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.627435 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:26:59.630873 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:59.632601 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:26:59.632793 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "bdf8009b1c954199a2f3e88ff33ebf9f"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "e9e27a3bd6a1a09cf0b15afc43852bb5"
server_key_iv: "47c2de6673ff96bb3654eb446a12acdc"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.632936 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:59.657613 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:59.662030 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:59.662246 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:59.664121 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:59.664201 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:59.664259 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:59.664304 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:26:59 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:59.677129 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38995
I20260925 16:26:59.677276 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38995 every 8 connection(s)
I20260925 16:26:59.678526 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:26:59.682973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1579
I20260925 16:26:59.683138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:26:59.683379 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.665332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57269 (local address 127.0.98.62:37543)
0925 16:26:59.665536 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:59.665542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:59.667109 (+ 1567us) server_negotiation.cc:407] Connection header received
0925 16:26:59.668615 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:59.668621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:59.668712 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:59.668834 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:59.670850 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.671561 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.673528 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.673884 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.673992 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.678038 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:59.678044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:59.678047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:59.678091 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:59.680085 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.680929 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.680933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.680934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.681058 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.681613 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.681616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.681618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.681951 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:59.682051 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:59.682918 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 16:26:59.683193 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:26:59.683490 392 external_mini_cluster.cc:1428] Setting key c3c85011fc8b8ab6da9b70d669af019f
I20260925 16:26:59.684767 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.666022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.1:57269)
0925 16:26:59.666861 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:59.666943 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:59.668287 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:59.669106 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:59.669118 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:59.669699 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:59.670623 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.670648 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.671722 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.671726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.673315 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.673331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.674100 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.674105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.674363 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.675460 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:59.675487 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:59.677695 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:59.680288 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.680296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.680318 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.680708 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.681205 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.681208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.681210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.681449 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.682107 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:59.682114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:59.682518 (+ 404us) client_negotiation.cc:769] Sending connection context
0925 16:26:59.682813 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 16:26:59.683371 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":186,"threads_started":1}
I20260925 16:26:59.686744 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37543
I20260925 16:26:59.686970 1710 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:59.687781 1710 heartbeater.cc:511] Master 127.0.98.62:37543 requested a full tablet report, sending...
I20260925 16:26:59.691182 1507 ts_manager.cc:194] Registered new tserver with Master: bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1:38995)
I20260925 16:26:59.692838 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57269
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:26:59.700719 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.690227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58013 (local address 127.0.98.62:37543)
0925 16:26:59.690541 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:59.690547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:59.690643 (+ 96us) server_negotiation.cc:407] Connection header received
0925 16:26:59.690833 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:59.690837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:59.690901 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:59.691008 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:59.691746 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.692742 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.694187 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.694518 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.694648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.696382 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:59.696387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:59.696390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:59.696422 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:59.698144 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.698861 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.698865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.698867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.699025 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.699394 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.699398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.699400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.699843 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:59.700007 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:59.700390 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 16:26:59.700527 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":207}
I20260925 16:26:59.700743 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.690120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.1:58013)
0925 16:26:59.690515 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:59.690534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:59.690716 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:59.691062 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:59.691065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:59.691368 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:59.691585 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.691596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.692944 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.692949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.694019 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.694038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.694686 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.694690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.694930 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.695662 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:59.695681 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:59.696190 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:59.698366 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.698370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.698372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.698673 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.699169 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.699173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.699176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.699290 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.700040 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:59.700045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:59.700186 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 16:26:59.700318 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 16:26:59.700535 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":3,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:26:59.722141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:26:59.853602 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:26:59.854008 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:26:59.854090 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:26:59.858827 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:26:59.858935 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:26:59.858981 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:26:59.859045 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:26:59.865265 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:26:59.866539 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:26:59.867698 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:26:59.878996 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:59.879029 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:26:59.879029 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:26:59.879177 1719 server_base.cc:1034] running on GCE node
I20260925 16:26:59.879904 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:26:59.880800 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:26:59.881979 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790353619881960 us; error 45 us; skew 500 ppm
Sep 25 16:26:59 dist-test-slave-31kt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:26:59.885738 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:26:59.887331 1719 webserver.cc:460] Webserver started at http://127.0.98.2:40249/ using document root <none> and password file <none>
I20260925 16:26:59.888228 1719 fs_manager.cc:359] Metadata directory not provided
I20260925 16:26:59.888350 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:26:59.888787 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:26:59.892475 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d38c98e05204102a0e41c6c0631551d"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "c1136da3730a94492389b9a760a07748"
server_key_iv: "820ecca47ffc57d64c0c882d660e7b54"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.893363 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d38c98e05204102a0e41c6c0631551d"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "c1136da3730a94492389b9a760a07748"
server_key_iv: "820ecca47ffc57d64c0c882d660e7b54"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.899402 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260925 16:26:59.903417 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:59.905515 1719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:26:59.905861 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8d38c98e05204102a0e41c6c0631551d"
format_stamp: "Formatted at 2026-09-25 16:26:59 on dist-test-slave-31kt"
server_key: "c1136da3730a94492389b9a760a07748"
server_key_iv: "820ecca47ffc57d64c0c882d660e7b54"
server_key_version: "encryptionkey@0"
I20260925 16:26:59.906081 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:26:59.927881 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:26:59.931475 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:26:59.931665 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:26:59.933548 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:26:59.933630 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:59.933686 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:26:59.933722 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:26:59.945338 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38037
I20260925 16:26:59.945362 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38037 every 8 connection(s)
I20260925 16:26:59.946507 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 16:26:59 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:26:59.951332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1719
I20260925 16:26:59.951469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:26:59.952009 392 external_mini_cluster.cc:1428] Setting key eb3947895920be6309a3938d4a8a5d62
I20260925 16:26:59.952191 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.934879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59033 (local address 127.0.98.62:37543)
0925 16:26:59.935077 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:59.935082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:59.936462 (+ 1380us) server_negotiation.cc:407] Connection header received
0925 16:26:59.938164 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:59.938169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:59.938244 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:59.938306 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:59.940474 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.941377 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.943425 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.943820 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.943990 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.947562 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:59.947566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:59.947568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:59.947603 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:59.949051 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.949931 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.949935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.949936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.950051 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.950509 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.950512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.950514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.950768 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:59.950856 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:59.951812 (+ 956us) server_negotiation.cc:299] Negotiation successful
0925 16:26:59.952025 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:26:59.953346 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.935449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.2:59033)
0925 16:26:59.936211 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:59.936293 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:59.937800 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:59.938838 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:59.938849 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:59.939419 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:59.940208 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.940231 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.941587 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.941609 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.943225 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.943243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.944077 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.944083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.944333 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.945233 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:59.945259 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:59.947250 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:59.949227 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.949235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.949254 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.949670 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.950167 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.950169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.950171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.950382 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.950882 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:59.950890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:59.951320 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 16:26:59.951554 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 16:26:59.951997 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":210,"threads_started":1}
I20260925 16:26:59.955077 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37543
I20260925 16:26:59.955333 1850 heartbeater.cc:464] Registering TS with master...
I20260925 16:26:59.956211 1850 heartbeater.cc:511] Master 127.0.98.62:37543 requested a full tablet report, sending...
I20260925 16:26:59.957824 1502 ts_manager.cc:194] Registered new tserver with Master: 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037)
I20260925 16:26:59.958647 1502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59033
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:26:59.969213 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.958516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46739 (local address 127.0.98.62:37543)
0925 16:26:59.958671 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:26:59.958676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:26:59.958842 (+ 166us) server_negotiation.cc:407] Connection header received
0925 16:26:59.959056 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:26:59.959060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:26:59.959120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:26:59.959175 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:26:59.960001 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.960895 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.962379 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.962644 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.962759 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.964617 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:26:59.964622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:26:59.964625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:26:59.964671 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:26:59.966593 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.967390 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.967393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.967395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.967519 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:26:59.967921 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:26:59.967924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:26:59.967926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:26:59.968212 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:26:59.968324 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:26:59.968837 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 16:26:59.969013 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 16:26:59.969225 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:26:59.958098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.2:46739)
0925 16:26:59.958715 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 16:26:59.958734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:26:59.958910 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:26:59.959361 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:26:59.959364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:26:59.959682 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:26:59.959849 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.959858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.961104 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.961108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.962193 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:26:59.962209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:26:59.962806 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:26:59.962811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:26:59.962934 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:26:59.963695 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:26:59.963714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:26:59.964276 (+ 562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:26:59.966862 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.966868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.966871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.967216 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.967628 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:26:59.967630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:26:59.967632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:26:59.967726 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:26:59.968383 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:26:59.968386 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:26:59.968537 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 16:26:59.968680 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 16:26:59.968936 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:26:59.990038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:00.135609 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:00.136053 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:00.136111 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:00.141072 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:00.141197 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:00.141249 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:00.141294 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:00.147877 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:00.149565 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:00.151070 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:00.164510 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:00.164528 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:00.164512 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:00.164974 1859 server_base.cc:1034] running on GCE node
I20260925 16:27:00.165581 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:00.166493 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:00.167749 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790353620167687 us; error 96 us; skew 500 ppm
Sep 25 16:27:00 dist-test-slave-31kt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:00.171443 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:00.172681 1859 webserver.cc:460] Webserver started at http://127.0.98.3:38655/ using document root <none> and password file <none>
I20260925 16:27:00.173422 1859 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:00.173507 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:00.173732 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:00.176533 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "ea345926ea7340988449638b6a1a4dd3"
format_stamp: "Formatted at 2026-09-25 16:27:00 on dist-test-slave-31kt"
server_key: "9d1af710ad925a485bfbe6599eb6d461"
server_key_iv: "9bdd9e72237c1189445a677e0cda68e5"
server_key_version: "encryptionkey@0"
I20260925 16:27:00.177255 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ea345926ea7340988449638b6a1a4dd3"
format_stamp: "Formatted at 2026-09-25 16:27:00 on dist-test-slave-31kt"
server_key: "9d1af710ad925a485bfbe6599eb6d461"
server_key_iv: "9bdd9e72237c1189445a677e0cda68e5"
server_key_version: "encryptionkey@0"
I20260925 16:27:00.182432 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:27:00.186378 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:00.188458 1859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:00.188656 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "ea345926ea7340988449638b6a1a4dd3"
format_stamp: "Formatted at 2026-09-25 16:27:00 on dist-test-slave-31kt"
server_key: "9d1af710ad925a485bfbe6599eb6d461"
server_key_iv: "9bdd9e72237c1189445a677e0cda68e5"
server_key_version: "encryptionkey@0"
I20260925 16:27:00.188798 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:00.209545 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:00.213348 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:00.213558 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:00.215562 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:00.215631 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:00.215691 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:00.215719 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:00.230129 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43021
Sep 25 16:27:00 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:00.231388 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:00.238683 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.217001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55653 (local address 127.0.98.62:37543)
0925 16:27:00.217206 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.217212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.218594 (+ 1382us) server_negotiation.cc:407] Connection header received
0925 16:27:00.220228 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.220233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.220329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.220484 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:00.222852 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.224216 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.227894 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.228300 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.228388 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.232349 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:00.232354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:00.232358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:00.232394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:00.234582 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.235575 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.235580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.235582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.235704 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.236170 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.236174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.236176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.236575 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:00.236706 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.238214 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.238449 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 16:27:00.240103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 1859
I20260925 16:27:00.240255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:00.240636 392 external_mini_cluster.cc:1428] Setting key b730dd3a87b8706271d1cc73b49cfe4b
I20260925 16:27:00.241904 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.217534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.3:55653)
0925 16:27:00.218376 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.218448 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.219848 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.220873 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.220886 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.221477 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:00.222385 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.222412 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.225947 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.225953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.227639 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.227661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.228697 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.228701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.228943 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.229688 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:00.229713 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:00.232073 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:00.234871 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:00.234881 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:00.234911 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:00.235423 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:00.235817 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:00.235820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:00.235823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:00.236054 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:00.236874 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:00.236894 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:00.237618 (+ 724us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.239469 (+ 1851us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.240140 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":287,"threads_started":1}
I20260925 16:27:00.240758 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43021 every 8 connection(s)
I20260925 16:27:00.243713 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37543
I20260925 16:27:00.243896 1990 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:00.244556 1990 heartbeater.cc:511] Master 127.0.98.62:37543 requested a full tablet report, sending...
I20260925 16:27:00.245959 1507 ts_manager.cc:194] Registered new tserver with Master: ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021)
I20260925 16:27:00.246812 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55653
I20260925 16:27:00.255970 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.246844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.3:48075)
0925 16:27:00.247299 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.247314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.247437 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.247707 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.247710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.247957 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:00.248096 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.248102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.248915 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.248918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.249551 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.249560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.250089 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.250093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.250194 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.250969 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:00.250983 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:00.251461 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:00.253548 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:00.253551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:00.253553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:00.253867 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:00.254167 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:00.254171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:00.254173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:00.254263 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:00.254865 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:00.254871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:00.255070 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.255262 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.255652 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":193,"threads_started":1}
I20260925 16:27:00.256119 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.246689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48075 (local address 127.0.98.62:37543)
0925 16:27:00.246892 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.246898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.247399 (+ 501us) server_negotiation.cc:407] Connection header received
0925 16:27:00.247561 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.247563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.247607 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.247673 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:00.248204 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.248786 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.249663 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.249966 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.250057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.251664 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:00.251667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:00.251670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:00.251696 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:00.253365 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.253984 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.253989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.253990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.254077 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.254355 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.254358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.254360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.254679 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:00.254784 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.255666 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.255840 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:27:00.256682 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:27:00 dist-test-slave-31kt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:27:00 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:00.287941 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.276952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.0.98.62:37543)
0925 16:27:00.277138 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.277142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.277222 (+ 80us) server_negotiation.cc:407] Connection header received
0925 16:27:00.277459 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.277462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.277531 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.277643 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:00.278716 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.279381 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.280496 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.280756 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.280859 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.283723 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:00.283727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:00.283731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:00.283775 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:00.285458 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.286181 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.286185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.286187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.286293 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.286662 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.286664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.286666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.286921 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:00.287057 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.287591 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.287735 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 16:27:00.292385 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:00.296587 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:00.318317 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.309515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43021 (local address 127.0.0.1:49772)
0925 16:27:00.312161 (+ 2646us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.312177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.312269 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.313205 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.313209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.313232 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.313352 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.313359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.314694 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.314697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.315785 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.315796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.317274 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.317279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.317516 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.317575 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.317803 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.318090 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2572,"mutex_wait_us":1018,"thread_start_us":116,"threads_started":1}
I20260925 16:27:00.318367 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.309887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.0.98.3:43021)
0925 16:27:00.312639 (+ 2752us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.312656 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.312685 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:27:00.312767 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.312771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.312973 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.313217 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.313464 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.314547 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.315962 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.317099 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.317224 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.317471 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.317967 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.318163 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2616,"thread_start_us":163,"threads_started":1}
I20260925 16:27:00.319211 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.309852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38995 (local address 127.0.0.1:59622)
0925 16:27:00.310718 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.310804 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.310979 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.311860 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.311867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.311895 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.312094 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.312103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.314246 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.314249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.315763 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.315776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.317274 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.317279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.318851 (+ 1572us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.318880 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.318972 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.319069 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":1437,"thread_start_us":137,"threads_started":1}
I20260925 16:27:00.319387 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.310766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.0.98.1:38995)
0925 16:27:00.311221 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.311231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.311260 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:27:00.311364 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.311371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.311684 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.311866 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.312251 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.313585 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.315930 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.317075 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.317224 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.317448 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.319074 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.319240 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":123,"threads_started":1}
I20260925 16:27:00.322405 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 182037009c2d49b6a38ec3e0ef02a39b (DEFAULT_TABLE table=test-table [id=cfd60bdb31684439b5afb5ac197b3fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:00.323077 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.309515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.0.1:53606)
0925 16:27:00.310727 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.310799 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.311052 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.312092 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.312096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.312123 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.312278 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.312286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.313734 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.313738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.315775 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.315782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.322455 (+ 6673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.322461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.322616 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.322656 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.322784 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.322891 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":181,"thread_start_us":109,"threads_started":1}
I20260925 16:27:00.323670 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.310044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.0.98.2:38037)
0925 16:27:00.311407 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.311422 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.311450 (+ 28us) server_negotiation.cc:407] Connection header received
0925 16:27:00.311541 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.311548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.311800 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.312132 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.312431 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.313597 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.317556 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.322199 (+ 4643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.322355 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.322556 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.323331 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.323491 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":130,"threads_started":1}
I20260925 16:27:00.324255 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182037009c2d49b6a38ec3e0ef02a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.326921 1766 tablet_service.cc:1467] Processing CreateTablet for tablet 182037009c2d49b6a38ec3e0ef02a39b (DEFAULT_TABLE table=test-table [id=cfd60bdb31684439b5afb5ac197b3fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:00.326920 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 182037009c2d49b6a38ec3e0ef02a39b (DEFAULT_TABLE table=test-table [id=cfd60bdb31684439b5afb5ac197b3fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:00.328994 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182037009c2d49b6a38ec3e0ef02a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.329239 1766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182037009c2d49b6a38ec3e0ef02a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.333570 2013 tablet_bootstrap.cc:492] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Bootstrap starting.
I20260925 16:27:00.337285 2013 tablet_bootstrap.cc:654] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.338434 2013 log.cc:824] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:00.339462 2014 tablet_bootstrap.cc:492] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Bootstrap starting.
I20260925 16:27:00.339803 2015 tablet_bootstrap.cc:492] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Bootstrap starting.
I20260925 16:27:00.341413 2013 tablet_bootstrap.cc:492] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: No bootstrap required, opened a new log
I20260925 16:27:00.341677 2013 ts_tablet_manager.cc:1397] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:27:00.342492 2015 tablet_bootstrap.cc:654] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.342605 2014 tablet_bootstrap.cc:654] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.343590 2015 log.cc:824] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:00.343739 2014 log.cc:824] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:00.346650 2014 tablet_bootstrap.cc:492] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: No bootstrap required, opened a new log
I20260925 16:27:00.346649 2015 tablet_bootstrap.cc:492] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: No bootstrap required, opened a new log
I20260925 16:27:00.346596 2013 raft_consensus.cc:348] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.346938 2014 ts_tablet_manager.cc:1397] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 16:27:00.346938 2015 ts_tablet_manager.cc:1397] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:27:00.346968 2013 raft_consensus.cc:374] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.347031 2013 raft_consensus.cc:729] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea345926ea7340988449638b6a1a4dd3, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.347615 2013 consensus_queue.cc:260] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.348533 1990 heartbeater.cc:503] Master 127.0.98.62:37543 was elected leader, sending a full tablet report...
I20260925 16:27:00.349056 2013 ts_tablet_manager.cc:1428] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:00.351905 2015 raft_consensus.cc:348] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.351905 2014 raft_consensus.cc:348] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.352303 2014 raft_consensus.cc:374] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.352303 2015 raft_consensus.cc:374] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.352360 2014 raft_consensus.cc:729] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf8009b1c954199a2f3e88ff33ebf9f, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.352360 2015 raft_consensus.cc:729] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d38c98e05204102a0e41c6c0631551d, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.353003 2014 consensus_queue.cc:260] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.353003 2015 consensus_queue.cc:260] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.354385 2015 ts_tablet_manager.cc:1428] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:00.354385 2014 ts_tablet_manager.cc:1428] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:00.354640 1850 heartbeater.cc:503] Master 127.0.98.62:37543 was elected leader, sending a full tablet report...
I20260925 16:27:00.354899 1710 heartbeater.cc:503] Master 127.0.98.62:37543 was elected leader, sending a full tablet report...
I20260925 16:27:00.372598 2020 raft_consensus.cc:482] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:00.372849 2020 raft_consensus.cc:504] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.375010 2020 leader_election.cc:288] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021), 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037)
I20260925 16:27:00.382515 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.375630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58205 (local address 127.0.98.3:43021)
0925 16:27:00.375807 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.375812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.376186 (+ 374us) server_negotiation.cc:407] Connection header received
0925 16:27:00.376396 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.376400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.376475 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.376569 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.377244 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.378287 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.380608 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.381366 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.381527 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.381623 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.382204 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.382328 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 16:27:00.382515 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.375595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37499 (local address 127.0.98.2:38037)
0925 16:27:00.375778 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.375783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.376233 (+ 450us) server_negotiation.cc:407] Connection header received
0925 16:27:00.376396 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.376400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.376475 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.376569 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.377222 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.378290 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.380608 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.381364 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.381527 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.381644 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.382204 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.382327 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:27:00.382566 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.375394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43021 (local address 127.0.98.1:58205)
0925 16:27:00.376011 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.376029 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.376226 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.376613 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.376616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.376664 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.377019 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.377028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.378527 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.378538 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.380322 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.380345 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.381624 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.381631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.381927 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.382014 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.382136 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.382299 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":304,"spinlock_wait_cycles":37504,"thread_start_us":238,"threads_started":1}
I20260925 16:27:00.382575 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.375394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.98.1:37499)
0925 16:27:00.376107 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.376119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.376225 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.376619 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.376622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.376665 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.377019 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.377028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.378527 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.378532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.380322 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.380345 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.381624 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.381631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.381957 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.382021 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.382136 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.382299 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":533,"spinlock_wait_cycles":14848,"thread_start_us":152,"threads_started":1}
I20260925 16:27:00.383543 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182037009c2d49b6a38ec3e0ef02a39b" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea345926ea7340988449638b6a1a4dd3" is_pre_election: true
I20260925 16:27:00.383877 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182037009c2d49b6a38ec3e0ef02a39b" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d38c98e05204102a0e41c6c0631551d" is_pre_election: true
I20260925 16:27:00.384011 1928 raft_consensus.cc:2454] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 0.
I20260925 16:27:00.384274 1804 raft_consensus.cc:2454] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 0.
I20260925 16:27:00.384809 1596 leader_election.cc:302] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdf8009b1c954199a2f3e88ff33ebf9f, ea345926ea7340988449638b6a1a4dd3; no voters:
I20260925 16:27:00.385231 2020 raft_consensus.cc:2781] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:00.385344 2020 raft_consensus.cc:482] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:00.385380 2020 raft_consensus.cc:3037] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.386943 2020 raft_consensus.cc:504] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.387720 2020 leader_election.cc:288] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 election: Requested vote from peers ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021), 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037)
I20260925 16:27:00.388403 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182037009c2d49b6a38ec3e0ef02a39b" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d38c98e05204102a0e41c6c0631551d"
I20260925 16:27:00.388489 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182037009c2d49b6a38ec3e0ef02a39b" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea345926ea7340988449638b6a1a4dd3"
I20260925 16:27:00.388576 1804 raft_consensus.cc:3037] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.388723 1928 raft_consensus.cc:3037] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.390262 1928 raft_consensus.cc:2454] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 1.
I20260925 16:27:00.390281 1804 raft_consensus.cc:2454] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 1.
I20260925 16:27:00.390759 1596 leader_election.cc:302] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdf8009b1c954199a2f3e88ff33ebf9f, ea345926ea7340988449638b6a1a4dd3; no voters:
I20260925 16:27:00.391021 2020 raft_consensus.cc:2781] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:00.391305 2020 raft_consensus.cc:686] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 LEADER]: Becoming Leader. State: Replica: bdf8009b1c954199a2f3e88ff33ebf9f, State: Running, Role: LEADER
I20260925 16:27:00.391824 2020 consensus_queue.cc:237] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } }
I20260925 16:27:00.396709 1507 catalog_manager.cc:5495] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } health_report { overall_health: HEALTHY } } }
W20260925 16:27:00.430655 1711 tablet.cc:2369] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:00.434875 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.430047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.0.98.1:38995)
0925 16:27:00.430254 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.430259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.430292 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:27:00.430570 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.430573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.430652 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.430773 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:00.431148 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.431742 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.432803 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.433045 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.433146 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.433622 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.434337 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.434445 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.434605 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.434686 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 16:27:00.443066 1804 raft_consensus.cc:1261] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Refusing update from remote peer bdf8009b1c954199a2f3e88ff33ebf9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:00.443066 1928 raft_consensus.cc:1261] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Refusing update from remote peer bdf8009b1c954199a2f3e88ff33ebf9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:00.444268 2024 consensus_queue.cc:1035] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:00.444530 2020 consensus_queue.cc:1035] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:27:00.448632 1851 tablet.cc:2369] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:00.454704 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333288429324095488 to 7333288429128994816. Current Snapshot: MvccSnapshot[applied={T|T < 7333288429324095488}]
I20260925 16:27:00.458091 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333288429324095488 to 7333288429128994816. Current Snapshot: MvccSnapshot[applied={T|T < 7333288429324095488}]
I20260925 16:27:00.459626 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333288429324095488 to 7333288429128994816. Current Snapshot: MvccSnapshot[applied={T|T < 7333288429324095488}]
I20260925 16:27:00.472608 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:00.472920 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:00.476508 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: Sending DeleteTablet for 3 replicas of tablet 182037009c2d49b6a38ec3e0ef02a39b
I20260925 16:27:00.477550 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 182037009c2d49b6a38ec3e0ef02a39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59622
I20260925 16:27:00.477654 1766 tablet_service.cc:1514] Processing DeleteTablet for tablet 182037009c2d49b6a38ec3e0ef02a39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53606
I20260925 16:27:00.477960 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 182037009c2d49b6a38ec3e0ef02a39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49772
I20260925 16:27:00.478392 2036 tablet_replica.cc:330] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: stopping tablet replica
I20260925 16:27:00.478763 2037 tablet_replica.cc:330] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: stopping tablet replica
I20260925 16:27:00.478942 2036 raft_consensus.cc:2229] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:00.479034 2037 raft_consensus.cc:2229] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:00.479434 2036 raft_consensus.cc:2258] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:00.479434 2037 raft_consensus.cc:2258] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:00.479854 2038 tablet_replica.cc:330] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: stopping tablet replica
I20260925 16:27:00.479934 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:27:00.480147 2038 raft_consensus.cc:2229] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 LEADER]: Raft consensus shutting down.
W20260925 16:27:00.480600 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:00.480944 2038 raft_consensus.cc:2258] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:00.482991 2037 ts_tablet_manager.cc:1905] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:00.483287 2036 ts_tablet_manager.cc:1905] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:00.486726 2037 ts_tablet_manager.cc:1918] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:00.486889 2037 log.cc:1196] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/182037009c2d49b6a38ec3e0ef02a39b
I20260925 16:27:00.487413 2037 ts_tablet_manager.cc:1939] T 182037009c2d49b6a38ec3e0ef02a39b P ea345926ea7340988449638b6a1a4dd3: Deleting consensus metadata
I20260925 16:27:00.489035 1766 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5df451286a48b6add9116ab02a802d (DEFAULT_TABLE table=test-table [id=18daafa26ac642a6840342e5cf7ca861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:00.489494 1766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5df451286a48b6add9116ab02a802d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.489538 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5df451286a48b6add9116ab02a802d (DEFAULT_TABLE table=test-table [id=18daafa26ac642a6840342e5cf7ca861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:00.490044 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5df451286a48b6add9116ab02a802d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.493288 2015 tablet_bootstrap.cc:492] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Bootstrap starting.
I20260925 16:27:00.489116 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5df451286a48b6add9116ab02a802d (DEFAULT_TABLE table=test-table [id=18daafa26ac642a6840342e5cf7ca861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:00.494340 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5df451286a48b6add9116ab02a802d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.494629 2015 tablet_bootstrap.cc:654] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.496253 2015 tablet_bootstrap.cc:492] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: No bootstrap required, opened a new log
I20260925 16:27:00.496362 2015 ts_tablet_manager.cc:1397] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:00.496886 2015 raft_consensus.cc:348] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.497007 2015 raft_consensus.cc:374] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.497037 2015 raft_consensus.cc:729] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d38c98e05204102a0e41c6c0631551d, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.497221 2015 consensus_queue.cc:260] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.497603 2015 ts_tablet_manager.cc:1428] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:00.497844 2013 tablet_bootstrap.cc:492] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Bootstrap starting.
I20260925 16:27:00.499027 2014 tablet_bootstrap.cc:492] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Bootstrap starting.
I20260925 16:27:00.488116 2038 ts_tablet_manager.cc:1905] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:00.499692 2013 tablet_bootstrap.cc:654] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.500447 2014 tablet_bootstrap.cc:654] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.501456 1492 catalog_manager.cc:4841] TS ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021): tablet 182037009c2d49b6a38ec3e0ef02a39b (table test-table [id=cfd60bdb31684439b5afb5ac197b3fbf]) successfully deleted
I20260925 16:27:00.502241 2013 tablet_bootstrap.cc:492] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: No bootstrap required, opened a new log
I20260925 16:27:00.502364 2013 ts_tablet_manager.cc:1397] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260925 16:27:00.502890 2013 raft_consensus.cc:348] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.503022 2013 raft_consensus.cc:374] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.503052 2013 raft_consensus.cc:729] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea345926ea7340988449638b6a1a4dd3, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.503190 2013 consensus_queue.cc:260] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.503463 2013 ts_tablet_manager.cc:1428] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:27:00.506172 2038 ts_tablet_manager.cc:1918] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:00.506346 2038 log.cc:1196] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/182037009c2d49b6a38ec3e0ef02a39b
I20260925 16:27:00.506364 2036 ts_tablet_manager.cc:1918] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:00.506552 2036 log.cc:1196] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/182037009c2d49b6a38ec3e0ef02a39b
I20260925 16:27:00.506757 2038 ts_tablet_manager.cc:1939] T 182037009c2d49b6a38ec3e0ef02a39b P bdf8009b1c954199a2f3e88ff33ebf9f: Deleting consensus metadata
I20260925 16:27:00.506963 2036 ts_tablet_manager.cc:1939] T 182037009c2d49b6a38ec3e0ef02a39b P 8d38c98e05204102a0e41c6c0631551d: Deleting consensus metadata
I20260925 16:27:00.508332 1493 catalog_manager.cc:4841] TS bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1:38995): tablet 182037009c2d49b6a38ec3e0ef02a39b (table test-table [id=cfd60bdb31684439b5afb5ac197b3fbf]) successfully deleted
I20260925 16:27:00.508882 1491 catalog_manager.cc:4841] TS 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037): tablet 182037009c2d49b6a38ec3e0ef02a39b (table test-table [id=cfd60bdb31684439b5afb5ac197b3fbf]) successfully deleted
I20260925 16:27:00.509116 2014 tablet_bootstrap.cc:492] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: No bootstrap required, opened a new log
I20260925 16:27:00.509239 2014 ts_tablet_manager.cc:1397] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 16:27:00.509805 2014 raft_consensus.cc:348] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.509949 2014 raft_consensus.cc:374] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.509981 2014 raft_consensus.cc:729] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf8009b1c954199a2f3e88ff33ebf9f, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.510121 2014 consensus_queue.cc:260] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.510841 2014 ts_tablet_manager.cc:1428] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:00.518528 2024 raft_consensus.cc:482] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:00.518750 2024 raft_consensus.cc:504] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.519368 2024 leader_election.cc:288] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037), ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021)
I20260925 16:27:00.520059 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5df451286a48b6add9116ab02a802d" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d38c98e05204102a0e41c6c0631551d" is_pre_election: true
I20260925 16:27:00.520087 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5df451286a48b6add9116ab02a802d" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea345926ea7340988449638b6a1a4dd3" is_pre_election: true
I20260925 16:27:00.520275 1804 raft_consensus.cc:2454] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 0.
I20260925 16:27:00.520296 1928 raft_consensus.cc:2454] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 0.
I20260925 16:27:00.520809 1595 leader_election.cc:302] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d38c98e05204102a0e41c6c0631551d, bdf8009b1c954199a2f3e88ff33ebf9f; no voters:
I20260925 16:27:00.521093 2024 raft_consensus.cc:2781] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:00.521155 2024 raft_consensus.cc:482] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:00.521193 2024 raft_consensus.cc:3037] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.522951 2024 raft_consensus.cc:504] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.523497 2024 leader_election.cc:288] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 election: Requested vote from peers 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037), ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021)
I20260925 16:27:00.524199 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5df451286a48b6add9116ab02a802d" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea345926ea7340988449638b6a1a4dd3"
I20260925 16:27:00.524240 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5df451286a48b6add9116ab02a802d" candidate_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d38c98e05204102a0e41c6c0631551d"
I20260925 16:27:00.524410 1804 raft_consensus.cc:3037] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.524396 1928 raft_consensus.cc:3037] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.525933 1928 raft_consensus.cc:2454] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 1.
I20260925 16:27:00.525942 1804 raft_consensus.cc:2454] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf8009b1c954199a2f3e88ff33ebf9f in term 1.
I20260925 16:27:00.526503 1596 leader_election.cc:302] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdf8009b1c954199a2f3e88ff33ebf9f, ea345926ea7340988449638b6a1a4dd3; no voters:
I20260925 16:27:00.526929 2024 raft_consensus.cc:2781] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:00.527072 2024 raft_consensus.cc:686] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 LEADER]: Becoming Leader. State: Replica: bdf8009b1c954199a2f3e88ff33ebf9f, State: Running, Role: LEADER
I20260925 16:27:00.527266 2024 consensus_queue.cc:237] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.529968 1505 catalog_manager.cc:5495] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:27:00 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:00.580626 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.568068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.0.98.62:37543)
0925 16:27:00.568252 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.568258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.568593 (+ 335us) server_negotiation.cc:407] Connection header received
0925 16:27:00.568796 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.568799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.568866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.569113 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:00.570318 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.570998 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.572242 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.572533 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.572674 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.576183 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:00.576188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:00.576190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:00.576248 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:00.577855 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.578718 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.578723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.578724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.578835 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.579222 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.579228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.579229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.579528 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:00.579680 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.580201 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.580413 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":52}
I20260925 16:27:00.580638 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.567860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.0.1:39962)
0925 16:27:00.568407 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.568425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.568629 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.569135 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.569138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.569954 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:00.570117 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.570128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.571156 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.571160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.572016 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.572028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.572725 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.572728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.572835 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.573683 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:00.573700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:00.575845 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:00.578039 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:00.578056 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:00.578066 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:00.578528 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:00.578952 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:00.578955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:00.578958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:00.579055 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:00.579689 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:00.579699 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:00.579904 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.580058 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.580316 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":125,"threads_started":1}
I20260925 16:27:00.586601 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39962:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:27:00.587958 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:00.595234 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 90545de4233b48688c96b6ec5c3599d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ade877df70f40a3a75c45b220da3ad1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:00.595381 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 90545de4233b48688c96b6ec5c3599d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ade877df70f40a3a75c45b220da3ad1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:00.595381 1766 tablet_service.cc:1467] Processing CreateTablet for tablet 90545de4233b48688c96b6ec5c3599d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ade877df70f40a3a75c45b220da3ad1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:00.595800 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90545de4233b48688c96b6ec5c3599d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.595850 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90545de4233b48688c96b6ec5c3599d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.595842 1766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90545de4233b48688c96b6ec5c3599d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:00.599397 2013 tablet_bootstrap.cc:492] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: Bootstrap starting.
I20260925 16:27:00.599663 2014 tablet_bootstrap.cc:492] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: Bootstrap starting.
I20260925 16:27:00.600430 2013 tablet_bootstrap.cc:654] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.601861 2014 tablet_bootstrap.cc:654] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.602263 2015 tablet_bootstrap.cc:492] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: Bootstrap starting.
I20260925 16:27:00.603410 2015 tablet_bootstrap.cc:654] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:00.603796 2014 tablet_bootstrap.cc:492] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: No bootstrap required, opened a new log
I20260925 16:27:00.603915 2014 ts_tablet_manager.cc:1397] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:00.604472 2014 raft_consensus.cc:348] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.604669 2014 raft_consensus.cc:374] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.604704 2014 raft_consensus.cc:729] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf8009b1c954199a2f3e88ff33ebf9f, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.604866 2014 consensus_queue.cc:260] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.604930 2015 tablet_bootstrap.cc:492] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: No bootstrap required, opened a new log
I20260925 16:27:00.605047 2015 ts_tablet_manager.cc:1397] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:00.605155 2014 ts_tablet_manager.cc:1428] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:00.605270 2013 tablet_bootstrap.cc:492] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: No bootstrap required, opened a new log
I20260925 16:27:00.605357 2013 ts_tablet_manager.cc:1397] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 16:27:00.605413 2024 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.605559 2024 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:00.605624 2015 raft_consensus.cc:348] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.605827 2015 raft_consensus.cc:374] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.605876 2015 raft_consensus.cc:729] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d38c98e05204102a0e41c6c0631551d, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.605878 2013 raft_consensus.cc:348] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.605995 2013 raft_consensus.cc:374] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:00.606024 2013 raft_consensus.cc:729] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea345926ea7340988449638b6a1a4dd3, State: Initialized, Role: FOLLOWER
I20260925 16:27:00.606163 2013 consensus_queue.cc:260] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.606436 2013 ts_tablet_manager.cc:1428] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:00.606597 2015 consensus_queue.cc:260] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.606711 2019 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.606909 2019 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:00.607177 2015 ts_tablet_manager.cc:1428] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:00.607110 2021 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.607352 2021 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 16:27:00.741252 1991 tablet.cc:2369] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:00.768205 2021 raft_consensus.cc:482] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:00.768391 2021 raft_consensus.cc:504] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.769670 2021 leader_election.cc:288] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1:38995), ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021)
I20260925 16:27:00.774199 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.770251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58793 (local address 127.0.98.3:43021)
0925 16:27:00.770454 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.770459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.770635 (+ 176us) server_negotiation.cc:407] Connection header received
0925 16:27:00.770813 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.770816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.770863 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.770930 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.771217 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.771751 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.773055 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.773495 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.773587 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.773666 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.773963 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.774042 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 16:27:00.774204 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.770191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34545 (local address 127.0.98.1:38995)
0925 16:27:00.770492 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.770497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.770801 (+ 304us) server_negotiation.cc:407] Connection header received
0925 16:27:00.770963 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.770966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.771016 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.771084 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:00.771356 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.771904 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.773061 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.773486 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.773589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.773674 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.773929 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.774018 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260925 16:27:00.774228 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.770141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38995 (local address 127.0.98.2:34545)
0925 16:27:00.770690 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.770703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.770848 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.771102 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.771105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.771120 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.771260 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.771265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.772018 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.772021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.772936 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.772945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.773613 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.773617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.773805 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.773832 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.773907 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.774018 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":3,"thread_start_us":194,"threads_started":1}
I20260925 16:27:00.774317 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.770068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43021 (local address 127.0.98.2:58793)
0925 16:27:00.770527 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.770542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.770692 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.770964 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.770968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.770988 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:00.771115 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.771122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.771855 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.771859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.772919 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.772931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.773609 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.773613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.773823 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.773861 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.773925 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.774033 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":69,"thread_start_us":121,"threads_started":1}
I20260925 16:27:00.775136 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90545de4233b48688c96b6ec5c3599d5" candidate_uuid: "8d38c98e05204102a0e41c6c0631551d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea345926ea7340988449638b6a1a4dd3" is_pre_election: true
I20260925 16:27:00.775199 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90545de4233b48688c96b6ec5c3599d5" candidate_uuid: "8d38c98e05204102a0e41c6c0631551d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" is_pre_election: true
I20260925 16:27:00.775346 1928 raft_consensus.cc:2454] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d38c98e05204102a0e41c6c0631551d in term 0.
I20260925 16:27:00.775489 1664 raft_consensus.cc:2454] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d38c98e05204102a0e41c6c0631551d in term 0.
I20260925 16:27:00.775794 1736 leader_election.cc:302] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d38c98e05204102a0e41c6c0631551d, ea345926ea7340988449638b6a1a4dd3; no voters:
I20260925 16:27:00.776154 2021 raft_consensus.cc:2781] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:00.776230 2021 raft_consensus.cc:482] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:00.776279 2021 raft_consensus.cc:3037] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.777318 2021 raft_consensus.cc:504] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.777726 2021 leader_election.cc:288] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [CANDIDATE]: Term 1 election: Requested vote from peers bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1:38995), ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021)
I20260925 16:27:00.778257 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90545de4233b48688c96b6ec5c3599d5" candidate_uuid: "8d38c98e05204102a0e41c6c0631551d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea345926ea7340988449638b6a1a4dd3"
I20260925 16:27:00.778254 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90545de4233b48688c96b6ec5c3599d5" candidate_uuid: "8d38c98e05204102a0e41c6c0631551d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f"
I20260925 16:27:00.778432 1928 raft_consensus.cc:3037] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.778432 1664 raft_consensus.cc:3037] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:00.779922 1664 raft_consensus.cc:2454] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d38c98e05204102a0e41c6c0631551d in term 1.
I20260925 16:27:00.779922 1928 raft_consensus.cc:2454] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d38c98e05204102a0e41c6c0631551d in term 1.
I20260925 16:27:00.780297 1736 leader_election.cc:302] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d38c98e05204102a0e41c6c0631551d, ea345926ea7340988449638b6a1a4dd3; no voters:
I20260925 16:27:00.780565 2021 raft_consensus.cc:2781] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:00.781291 2021 raft_consensus.cc:686] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [term 1 LEADER]: Becoming Leader. State: Replica: 8d38c98e05204102a0e41c6c0631551d, State: Running, Role: LEADER
I20260925 16:27:00.781615 2021 consensus_queue.cc:237] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } }
I20260925 16:27:00.783159 2054 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: TxnStatusTablet state changed. Reason: New leader 8d38c98e05204102a0e41c6c0631551d. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.783311 2054 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:00.783684 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:27:00.783896 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:27:00.784906 1505 catalog_manager.cc:5495] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d38c98e05204102a0e41c6c0631551d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:00.815395 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.810026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.0.1:53616)
0925 16:27:00.810262 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.810282 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.810441 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.810914 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.810916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.810928 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:00.811062 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.811067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.812145 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.812148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.812909 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.812921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.813617 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.813620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.813722 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.813863 (+ 141us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.814981 (+ 1118us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.815006 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.815089 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.815225 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 16:27:00.815428 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.810168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.98.2:38037)
0925 16:27:00.810400 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.810406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.810432 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:27:00.810663 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.810667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.810750 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.810893 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:00.811179 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.811975 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.813057 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.813437 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.813559 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.814069 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.814784 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.814930 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.815125 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.815209 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 16:27:00.819895 1928 raft_consensus.cc:1261] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Refusing update from remote peer 8d38c98e05204102a0e41c6c0631551d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:00.820111 1664 raft_consensus.cc:1261] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Refusing update from remote peer 8d38c98e05204102a0e41c6c0631551d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:00.820631 2054 consensus_queue.cc:1035] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:00.820878 2021 consensus_queue.cc:1035] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:00.823140 2019 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: TxnStatusTablet state changed. Reason: New leader 8d38c98e05204102a0e41c6c0631551d. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.823323 2019 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:00.823601 2024 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: TxnStatusTablet state changed. Reason: New leader 8d38c98e05204102a0e41c6c0631551d. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.823731 2024 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:00.825258 2021 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.825392 2021 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:00.826418 2019 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.826622 2019 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P ea345926ea7340988449638b6a1a4dd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:00.830626 2054 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.830816 2054 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P 8d38c98e05204102a0e41c6c0631551d: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:00.831933 2024 tablet_replica.cc:439] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d38c98e05204102a0e41c6c0631551d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf8009b1c954199a2f3e88ff33ebf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38995 } } peers { permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 } } }
I20260925 16:27:00.832111 2024 tablet_replica.cc:442] T 90545de4233b48688c96b6ec5c3599d5 P bdf8009b1c954199a2f3e88ff33ebf9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:00.847846 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.843951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.98.1:45637)
0925 16:27:00.844386 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.844402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.844604 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.844869 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.844872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.844882 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:00.845013 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.845019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.845858 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.845862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.846548 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.846557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.846999 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.847003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.847091 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.847166 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.847495 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.847503 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.847563 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.847683 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":117,"threads_started":1}
I20260925 16:27:00.847844 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.844028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45637 (local address 127.0.98.2:38037)
0925 16:27:00.844203 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.844208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.844545 (+ 337us) server_negotiation.cc:407] Connection header received
0925 16:27:00.844724 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.844728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.844784 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.844851 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:00.845112 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.845714 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.846657 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.846885 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.846984 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.847313 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.847415 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.847473 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.847580 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.847641 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:27:00.856675 1936 raft_consensus.cc:1261] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Refusing update from remote peer bdf8009b1c954199a2f3e88ff33ebf9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:00.857172 1804 raft_consensus.cc:1261] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Refusing update from remote peer bdf8009b1c954199a2f3e88ff33ebf9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:00.857647 2024 consensus_queue.cc:1035] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea345926ea7340988449638b6a1a4dd3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:00.857896 2020 consensus_queue.cc:1035] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d38c98e05204102a0e41c6c0631551d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:00.889636 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.884564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40179 (local address 127.0.98.1:38995)
0925 16:27:00.884711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.884717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.885057 (+ 340us) server_negotiation.cc:407] Connection header received
0925 16:27:00.885181 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.885185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.885268 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.885373 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:00.885743 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.886652 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.888024 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.888445 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.888568 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.889014 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.889141 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.889213 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.889340 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.889426 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:27:00.889655 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.884482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38995 (local address 127.0.98.2:40179)
0925 16:27:00.884934 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:00.884948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:00.885070 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:00.885447 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:00.885451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:00.885461 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:00.885619 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.885625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.886793 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.886798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.887887 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:00.887899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.888600 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.888604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:00.888684 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.888834 (+ 150us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:00.889227 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:00.889244 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 16:27:00.889299 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 16:27:00.889426 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":142,"threads_started":1}
I20260925 16:27:00.915937 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:00.916155 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:00.918747 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2505cb233244cd995d19605add56638: Sending DeleteTablet for 3 replicas of tablet 7b5df451286a48b6add9116ab02a802d
I20260925 16:27:00.919559 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5df451286a48b6add9116ab02a802d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59622
I20260925 16:27:00.919579 1766 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5df451286a48b6add9116ab02a802d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53606
I20260925 16:27:00.919839 2036 tablet_replica.cc:330] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: stopping tablet replica
I20260925 16:27:00.919839 2038 tablet_replica.cc:330] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: stopping tablet replica
I20260925 16:27:00.920003 2036 raft_consensus.cc:2229] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:00.920009 2038 raft_consensus.cc:2229] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:00.920159 2036 raft_consensus.cc:2258] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:00.920292 2038 raft_consensus.cc:2258] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:00.920325 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5df451286a48b6add9116ab02a802d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49772
I20260925 16:27:00.920601 2037 tablet_replica.cc:330] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: stopping tablet replica
I20260925 16:27:00.920758 2037 raft_consensus.cc:2229] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:00.920900 2037 raft_consensus.cc:2258] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:00.921299 2036 ts_tablet_manager.cc:1905] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:00.921870 2037 ts_tablet_manager.cc:1905] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:00.924917 2037 ts_tablet_manager.cc:1918] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:00.925024 2037 log.cc:1196] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/7b5df451286a48b6add9116ab02a802d
I20260925 16:27:00.925014 2036 ts_tablet_manager.cc:1918] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:00.925102 2036 log.cc:1196] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/7b5df451286a48b6add9116ab02a802d
I20260925 16:27:00.925350 2037 ts_tablet_manager.cc:1939] T 7b5df451286a48b6add9116ab02a802d P ea345926ea7340988449638b6a1a4dd3: Deleting consensus metadata
I20260925 16:27:00.925376 2036 ts_tablet_manager.cc:1939] T 7b5df451286a48b6add9116ab02a802d P 8d38c98e05204102a0e41c6c0631551d: Deleting consensus metadata
I20260925 16:27:00.926482 2038 ts_tablet_manager.cc:1905] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:00.926687 1492 catalog_manager.cc:4841] TS ea345926ea7340988449638b6a1a4dd3 (127.0.98.3:43021): tablet 7b5df451286a48b6add9116ab02a802d (table test-table [id=18daafa26ac642a6840342e5cf7ca861]) successfully deleted
I20260925 16:27:00.926910 1491 catalog_manager.cc:4841] TS 8d38c98e05204102a0e41c6c0631551d (127.0.98.2:38037): tablet 7b5df451286a48b6add9116ab02a802d (table test-table [id=18daafa26ac642a6840342e5cf7ca861]) successfully deleted
I20260925 16:27:00.929653 2038 ts_tablet_manager.cc:1918] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:00.929730 2038 log.cc:1196] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/7b5df451286a48b6add9116ab02a802d
I20260925 16:27:00.929972 2038 ts_tablet_manager.cc:1939] T 7b5df451286a48b6add9116ab02a802d P bdf8009b1c954199a2f3e88ff33ebf9f: Deleting consensus metadata
I20260925 16:27:00.930922 1493 catalog_manager.cc:4841] TS bdf8009b1c954199a2f3e88ff33ebf9f (127.0.98.1:38995): tablet 7b5df451286a48b6add9116ab02a802d (table test-table [id=18daafa26ac642a6840342e5cf7ca861]) successfully deleted
Sep 25 16:27:00 dist-test-slave-31kt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:27:00.938037 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.921934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.98.1:38995)
0925 16:27:00.922112 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.922117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.922746 (+ 629us) server_negotiation.cc:407] Connection header received
0925 16:27:00.922899 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.922903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.922986 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.923092 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:00.924007 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.924792 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.927174 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.927555 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.927648 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.932167 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:00.932186 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:00.932205 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:00.932247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:00.935057 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.935836 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.935845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.935847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.935998 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.936488 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.936491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.936492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.936922 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:00.937163 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.937537 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.937796 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 16:27:00.938663 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59640
I20260925 16:27:00.952819 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:00.941163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.0.98.62:37543)
0925 16:27:00.941337 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:00.941341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:00.941363 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:27:00.941524 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:00.941527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:00.941587 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:00.941715 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:00.942668 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.943574 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.944810 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:00.945233 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:00.945377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:00.947749 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:00.947755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:00.947759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:00.947819 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:00.950085 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.950865 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.950871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.950873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.951059 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:00.951454 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:00.951458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:00.951460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:00.951760 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:00.951946 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:00.952277 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 16:27:00.952466 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 16:27:00.953476 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39976
I20260925 16:27:00.955626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1579
I20260925 16:27:00.967828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1719
I20260925 16:27:00.976490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1859
I20260925 16:27:00.982893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 1475
2026-09-25T16:27:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3137 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:01 dist-test-slave-31kt krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:01 dist-test-slave-31kt krb5kdc[2078](info): set up 2 sockets
Sep 25 16:27:01 dist-test-slave-31kt krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260925 16:27:01.970121 392 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:02 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:02.113600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:02.256798 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:02.257321 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:02.257393 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:02.262452 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:02.262549 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:27:02.262574 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:02.262598 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:02.262620 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:02.262645 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:02.269636 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:02.271354 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:02.272837 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:02.284305 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:02.284325 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:02.284325 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:02.284611 2091 server_base.cc:1034] running on GCE node
I20260925 16:27:02.285430 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:02.286979 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:02.290360 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790353622290297 us; error 84 us; skew 500 ppm
Sep 25 16:27:02 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:02.293993 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:02.295377 2091 webserver.cc:460] Webserver started at http://127.0.98.62:36619/ using document root <none> and password file <none>
I20260925 16:27:02.296157 2091 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:02.296244 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:02.296505 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:02.299137 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "031e75bd4a8048dfa9b386fce17e6f78"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "625be51bd7e53e4e910cbf7b9833b7db"
server_key_iv: "fbcd467025f8834fdf352586329f8220"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.299803 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "031e75bd4a8048dfa9b386fce17e6f78"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "625be51bd7e53e4e910cbf7b9833b7db"
server_key_iv: "fbcd467025f8834fdf352586329f8220"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.305130 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:02.308830 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:02.311067 2091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:27:02.311273 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "031e75bd4a8048dfa9b386fce17e6f78"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "625be51bd7e53e4e910cbf7b9833b7db"
server_key_iv: "fbcd467025f8834fdf352586329f8220"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.311416 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:02.351884 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:02.356515 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:02.356763 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:02.366822 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36155
I20260925 16:27:02.366928 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36155 every 8 connection(s)
I20260925 16:27:02.368028 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:02.372247 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:02.373685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 2091
I20260925 16:27:02.373838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:02.374290 392 external_mini_cluster.cc:1428] Setting key 4871cf31fdcf1464bb269551b2199df1
I20260925 16:27:02.380728 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Bootstrap starting.
I20260925 16:27:02.384279 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:02.385243 2169 log.cc:824] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:27:02 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:02.388983 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: No bootstrap required, opened a new log
I20260925 16:27:02.393989 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031e75bd4a8048dfa9b386fce17e6f78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } }
I20260925 16:27:02.394379 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:02.394379 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.376172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.0.98.62:36155)
0925 16:27:02.376820 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:02.376841 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:02.376891 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:27:02.378080 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:02.378131 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:02.378701 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:02.379147 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:02.380075 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.381214 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.382160 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.382750 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.382918 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:02.386412 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:02.386424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:02.386451 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:02.386486 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:02.389885 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:02.390616 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:02.390625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:02.390633 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:02.390831 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:02.391215 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:02.391219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:02.391221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:02.391925 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:02.392165 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:02.392536 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 16:27:02.392993 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":194,"threads_started":1}
I20260925 16:27:02.394460 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031e75bd4a8048dfa9b386fce17e6f78, State: Initialized, Role: FOLLOWER
I20260925 16:27:02.395033 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031e75bd4a8048dfa9b386fce17e6f78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } }
I20260925 16:27:02.395211 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:02.395278 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:02.395385 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:02.396889 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031e75bd4a8048dfa9b386fce17e6f78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } }
I20260925 16:27:02.397339 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 031e75bd4a8048dfa9b386fce17e6f78; no voters:
I20260925 16:27:02.397830 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:02.397931 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:02.398131 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [term 1 LEADER]: Becoming Leader. State: Replica: 031e75bd4a8048dfa9b386fce17e6f78, State: Running, Role: LEADER
I20260925 16:27:02.398514 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031e75bd4a8048dfa9b386fce17e6f78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } }
I20260925 16:27:02.399508 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:02.400734 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "031e75bd4a8048dfa9b386fce17e6f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031e75bd4a8048dfa9b386fce17e6f78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } }
I20260925 16:27:02.400969 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:02.401381 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 031e75bd4a8048dfa9b386fce17e6f78. Latest consensus state: current_term: 1 leader_uuid: "031e75bd4a8048dfa9b386fce17e6f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "031e75bd4a8048dfa9b386fce17e6f78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } }
I20260925 16:27:02.401515 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:02.401639 2183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:02.405638 2183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:02.414175 2183 catalog_manager.cc:1347] Generated new cluster ID: 763cad1de4394537a31b50402a766c2f
I20260925 16:27:02.414289 2183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:02.454998 2183 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:02.456681 2183 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:02.471506 2183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Generated new TSK 0
I20260925 16:27:02.472847 2183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:27:02.505941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36155
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:02.643280 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:02.643714 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:02.643941 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:02.649555 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:02.649658 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:02.649709 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:27:02.649724 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:02.649789 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:02.657028 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:02.658861 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:02.660207 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:02.671752 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:02.671828 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:02.671752 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:02.672087 2195 server_base.cc:1034] running on GCE node
I20260925 16:27:02.672794 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:02.673570 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:02.674783 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790353622674746 us; error 61 us; skew 500 ppm
Sep 25 16:27:02 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:02.684423 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:02.685580 2195 webserver.cc:460] Webserver started at http://127.0.98.1:37147/ using document root <none> and password file <none>
I20260925 16:27:02.686251 2195 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:02.686326 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:02.686537 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:02.688769 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "4b814a01927caf00898f4c8b80e87f8f"
server_key_iv: "54083be037dbd509da47a25afaed06c5"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.689354 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "4b814a01927caf00898f4c8b80e87f8f"
server_key_iv: "54083be037dbd509da47a25afaed06c5"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.693925 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 16:27:02.697532 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:02.699612 2195 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:02.699785 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "4b814a01927caf00898f4c8b80e87f8f"
server_key_iv: "54083be037dbd509da47a25afaed06c5"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.700102 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:02.717403 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:02.722211 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:02.722420 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:02.724231 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:02.724298 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:02.724342 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:02.724356 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:02.735816 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33793
I20260925 16:27:02.736071 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33793 every 8 connection(s)
Sep 25 16:27:02 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:02.737191 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:02.742368 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.725891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48217 (local address 127.0.98.62:36155)
0925 16:27:02.726098 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:02.726104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:02.727306 (+ 1202us) server_negotiation.cc:407] Connection header received
0925 16:27:02.728759 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:02.728763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:02.728842 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:02.728926 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:02.730928 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.731574 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.733014 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.733282 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.733389 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:02.737190 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:02.737194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:02.737197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:02.737232 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:02.739207 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:02.740077 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:02.740081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:02.740083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:02.740202 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:02.740696 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:02.740700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:02.740702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:02.741047 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:02.741154 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:02.742038 (+ 884us) server_negotiation.cc:299] Negotiation successful
0925 16:27:02.742217 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:27:02.743520 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.726330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.98.1:48217)
0925 16:27:02.727106 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:02.727170 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:02.728483 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:02.729183 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:02.729195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:02.729853 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:02.730719 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:02.730747 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.731709 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.731714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:02.732871 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:02.732884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.733467 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.733472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:02.733738 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:02.734733 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:02.734758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:02.736908 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:02.739393 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:02.739400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:02.739418 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:02.739865 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:02.740326 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:02.740329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:02.740331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:02.740519 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:02.741188 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:02.741197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:02.741574 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 16:27:02.741853 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 16:27:02.742281 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"spinlock_wait_cycles":22912,"thread_start_us":147,"threads_started":1}
I20260925 16:27:02.745184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 2195
I20260925 16:27:02.745324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:02.745683 392 external_mini_cluster.cc:1428] Setting key 61ab602bb856852aa3a566a1aac255a5
I20260925 16:27:02.745790 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36155
I20260925 16:27:02.745990 2326 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:02.746711 2326 heartbeater.cc:511] Master 127.0.98.62:36155 requested a full tablet report, sending...
I20260925 16:27:02.748917 2123 ts_manager.cc:194] Registered new tserver with Master: cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793)
I20260925 16:27:02.750718 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48217
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:27:02.758919 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.749117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33931 (local address 127.0.98.62:36155)
0925 16:27:02.749263 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:02.749269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:02.749666 (+ 397us) server_negotiation.cc:407] Connection header received
0925 16:27:02.749816 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:02.749819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:02.749871 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:02.749965 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:02.750560 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.751330 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.752395 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.752941 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.753142 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:02.754742 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:02.754748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:02.754752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:02.754794 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:02.756699 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:02.757341 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:02.757346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:02.757348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:02.757460 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:02.757866 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:02.757869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:02.757871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:02.758136 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:02.758259 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:02.758656 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 16:27:02.758780 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:27:02.758944 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.749052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.98.1:33931)
0925 16:27:02.749554 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:02.749570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:02.749697 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:02.749998 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:02.750001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:02.750282 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:02.750426 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:02.750434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.751507 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.751510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:02.752247 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:02.752271 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:02.753161 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:02.753165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:02.753302 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:02.754050 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:02.754066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:02.754491 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:02.756890 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:02.756893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:02.756895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:02.757195 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:02.757602 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:02.757605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:02.757607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:02.757726 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:02.758305 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:02.758308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:02.758443 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 16:27:02.758546 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 16:27:02.758731 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":3,"thread_start_us":293,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:02.781185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36155
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:02.922713 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:02.923125 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:02.923204 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:02.928006 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:02.928119 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:02.928186 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:27:02.928216 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:02.928275 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:02.935348 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:02.937119 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:02.938671 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:27:02.951740 2335 server_base.cc:1034] running on GCE node
W20260925 16:27:02.951743 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:02.951700 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:02.952170 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:02.952766 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:02.953672 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:02.954864 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790353622954824 us; error 61 us; skew 500 ppm
Sep 25 16:27:02 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:02.958531 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:02.960093 2335 webserver.cc:460] Webserver started at http://127.0.98.2:38105/ using document root <none> and password file <none>
I20260925 16:27:02.961076 2335 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:02.961162 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:02.961438 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:02.964351 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "448470f3da4708ad20a92081d4b4c828"
server_key_iv: "4c0822c9955fc9d2ee923ab266299ad4"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.965137 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "448470f3da4708ad20a92081d4b4c828"
server_key_iv: "4c0822c9955fc9d2ee923ab266299ad4"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.970582 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 16:27:02.974216 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:02.976028 2335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:02.976193 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a"
format_stamp: "Formatted at 2026-09-25 16:27:02 on dist-test-slave-31kt"
server_key: "448470f3da4708ad20a92081d4b4c828"
server_key_iv: "4c0822c9955fc9d2ee923ab266299ad4"
server_key_version: "encryptionkey@0"
I20260925 16:27:02.976331 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:02.989113 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:02.993055 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:02.993276 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:02.995237 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:02.995306 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:02.995368 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:02.995395 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:27:03 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:03.007887 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33067
I20260925 16:27:03.008630 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33067 every 8 connection(s)
I20260925 16:27:03.009286 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:03.010075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 2335
I20260925 16:27:03.010207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:03.010524 392 external_mini_cluster.cc:1428] Setting key 6eae5ad9f06d22870a830aabfe9ee202
I20260925 16:27:03.013111 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.996536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37999 (local address 127.0.98.62:36155)
0925 16:27:02.996744 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:02.996749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:02.997887 (+ 1138us) server_negotiation.cc:407] Connection header received
0925 16:27:02.999304 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:02.999308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:02.999378 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:02.999455 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:03.001264 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.002027 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.003428 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.003738 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.003854 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.008044 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:03.008051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:03.008056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:03.008102 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:03.010349 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.011077 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.011081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.011083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.011176 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.011538 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.011541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.011543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.011790 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:03.011886 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.012527 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.012715 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 16:27:03.014766 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:02.997078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.98.2:37999)
0925 16:27:02.997673 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:02.997731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:02.998978 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:02.999719 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:02.999728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.000249 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:03.001087 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.001107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.002161 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.002165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.003280 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.003292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.004003 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.004011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.004347 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.005622 (+ 1275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:03.005650 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:03.007644 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:03.010568 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.010576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.010597 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.010951 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.011276 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.011279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.011281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.011439 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.011899 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:03.011908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:03.012246 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.012465 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.013179 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":145,"threads_started":1}
I20260925 16:27:03.016803 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36155
I20260925 16:27:03.017086 2466 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:03.018095 2466 heartbeater.cc:511] Master 127.0.98.62:36155 requested a full tablet report, sending...
I20260925 16:27:03.020125 2119 ts_manager.cc:194] Registered new tserver with Master: 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067)
I20260925 16:27:03.021088 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37999
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:03.031606 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.020067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54967 (local address 127.0.98.62:36155)
0925 16:27:03.020213 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.020218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.022494 (+ 2276us) server_negotiation.cc:407] Connection header received
0925 16:27:03.022728 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.022732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.022794 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.022860 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:03.023770 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.024383 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.025643 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.026018 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.026159 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.028136 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:03.028140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:03.028143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:03.028173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:03.029876 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.030363 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.030367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.030370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.030447 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.030695 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.030698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.030700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.030951 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:03.031040 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.031305 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.031425 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:27:03.031625 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.020098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.98.2:54967)
0925 16:27:03.022351 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.022371 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.022597 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.023022 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.023025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.023284 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:03.023472 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.023481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.024608 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.024612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.025413 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.025430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.026194 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.026197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.026309 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.027098 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:03.027117 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:03.027904 (+ 787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:03.030005 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.030008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.030011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.030261 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.030530 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.030534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.030536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.030613 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.031044 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:03.031048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:03.031152 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.031241 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.031417 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2122,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:03.046149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36155
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:03.187158 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:03.187564 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:03.187639 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:03.192260 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:03.192373 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:03.192437 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 16:27:03.192459 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:03.192523 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:03.199158 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:03.200834 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:03.202266 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:03.214167 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:03.214151 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:03.214151 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:03.214316 2475 server_base.cc:1034] running on GCE node
I20260925 16:27:03.215262 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:03.216405 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:03.217834 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790353623217802 us; error 62 us; skew 500 ppm
Sep 25 16:27:03 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:03.221393 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:03.222613 2475 webserver.cc:460] Webserver started at http://127.0.98.3:39357/ using document root <none> and password file <none>
I20260925 16:27:03.223295 2475 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:03.223363 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:03.223588 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:03.226052 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "fae69f4e3351498ba7cc458f3e8e5174"
format_stamp: "Formatted at 2026-09-25 16:27:03 on dist-test-slave-31kt"
server_key: "f949c4d77ad4a6c91cc8adc0b1d0d576"
server_key_iv: "d40feca662fd6dd91dfb50e7605b0341"
server_key_version: "encryptionkey@0"
I20260925 16:27:03.226660 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fae69f4e3351498ba7cc458f3e8e5174"
format_stamp: "Formatted at 2026-09-25 16:27:03 on dist-test-slave-31kt"
server_key: "f949c4d77ad4a6c91cc8adc0b1d0d576"
server_key_iv: "d40feca662fd6dd91dfb50e7605b0341"
server_key_version: "encryptionkey@0"
I20260925 16:27:03.231261 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:03.234612 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:03.236372 2475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:27:03.236557 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "fae69f4e3351498ba7cc458f3e8e5174"
format_stamp: "Formatted at 2026-09-25 16:27:03 on dist-test-slave-31kt"
server_key: "f949c4d77ad4a6c91cc8adc0b1d0d576"
server_key_iv: "d40feca662fd6dd91dfb50e7605b0341"
server_key_version: "encryptionkey@0"
I20260925 16:27:03.236706 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:03.256774 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:03.261054 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:03.261260 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:03.263123 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:03.263200 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:03.263257 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:03.263276 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:03.275146 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36419
I20260925 16:27:03.275170 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36419 every 8 connection(s)
Sep 25 16:27:03 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:03.276468 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:03.281790 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.264520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44289 (local address 127.0.98.62:36155)
0925 16:27:03.264676 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.264681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.266428 (+ 1747us) server_negotiation.cc:407] Connection header received
0925 16:27:03.268114 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.268119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.268207 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.268353 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:03.270407 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.271255 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.272918 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.273173 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.273289 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.277092 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:03.277099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:03.277102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:03.277150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:03.279204 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.279947 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.279951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.279953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.280050 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.280407 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.280410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.280412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.280666 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:03.280794 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.281404 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.281575 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:27:03.282793 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.265221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.98.3:44289)
0925 16:27:03.266163 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.266239 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.267746 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.268632 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.268646 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.269298 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:03.270142 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.270166 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.271438 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.271442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.272707 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.272730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.273355 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.273360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.273591 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.274408 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:03.274429 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:03.276741 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:03.279404 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.279412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.279433 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.279820 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.280141 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.280144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.280146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.280302 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.280792 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:03.280800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:03.281130 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.281321 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.281748 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":155,"threads_started":1}
I20260925 16:27:03.284536 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36155
I20260925 16:27:03.284785 2606 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:03.285380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 2475
I20260925 16:27:03.285506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:03.285652 2606 heartbeater.cc:511] Master 127.0.98.62:36155 requested a full tablet report, sending...
I20260925 16:27:03.285815 392 external_mini_cluster.cc:1428] Setting key d363eefd50fe8ce336e287ea9bfaff5c
I20260925 16:27:03.288161 2123 ts_manager.cc:194] Registered new tserver with Master: fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:03.289057 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44289
I20260925 16:27:03.297163 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.288250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43583 (local address 127.0.98.62:36155)
0925 16:27:03.288390 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.288396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.288861 (+ 465us) server_negotiation.cc:407] Connection header received
0925 16:27:03.288972 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.288974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.289029 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.289103 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:03.289791 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.290344 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.291319 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.291539 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.291638 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.292915 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:03.292919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:03.292921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:03.292945 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:03.294977 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.295536 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.295539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.295540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.295632 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.295904 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.295907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.295909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.296114 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:03.296205 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.296704 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.296897 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:27:03.297169 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.288176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.98.3:43583)
0925 16:27:03.288746 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.288763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.288885 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.289154 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.289157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.289438 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:03.289601 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.289611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.290515 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.290520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.291182 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.291193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.291667 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.291671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.291783 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.292336 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:03.292348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:03.292763 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:03.295173 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.295177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.295180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.295419 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.295728 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.295731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.295733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.295814 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.296209 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:03.296212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:03.296314 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.296563 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.296900 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":220,"threads_started":1}
I20260925 16:27:03.301241 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:27:03 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:27:03 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:03.333890 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.323467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.98.62:36155)
0925 16:27:03.323642 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.323647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.323662 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:27:03.323720 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.323725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.323775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.323860 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:03.324881 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.325531 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.326602 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.326851 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.326938 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.329887 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:03.329891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:03.329894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:03.329935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:03.331570 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.332163 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.332166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.332167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.332256 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.332754 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.332757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.332759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.333023 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:03.333159 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.333504 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.333639 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:27:03.338076 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:03.341675 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:03.361953 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.355051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33067 (local address 127.0.0.1:56444)
0925 16:27:03.355780 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.355859 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.356016 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.357145 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.357147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.357164 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.357370 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.357379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.358837 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.358840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.359999 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.360009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.361134 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.361139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.361291 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.361353 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.361521 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.361700 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":169,"thread_start_us":109,"threads_started":1}
I20260925 16:27:03.362624 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.355595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.0.98.2:33067)
0925 16:27:03.356340 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.356355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.356382 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:27:03.356479 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.356485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.356761 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.356953 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.357509 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.358709 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.360137 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.360923 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.361046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.361281 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.362311 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.362467 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":106,"threads_started":1}
I20260925 16:27:03.362982 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.354685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36419 (local address 127.0.0.1:56490)
0925 16:27:03.355780 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.355856 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.356011 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.356970 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.356976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.357010 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.357198 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.357206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.358808 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.358811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.360488 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.360507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.362558 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.362562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.362662 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.362689 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.362763 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.362845 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":96,"thread_start_us":104,"threads_started":1}
I20260925 16:27:03.363134 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.354913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.0.98.3:36419)
0925 16:27:03.356340 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.356356 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.356382 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:27:03.356479 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.356485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.356761 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.356953 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.357335 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.358677 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.361374 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.362405 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.362539 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.362748 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.362813 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.362938 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":193,"threads_started":1}
I20260925 16:27:03.364159 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.355649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.0.1:58822)
0925 16:27:03.355975 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.355987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.356066 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.357115 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.357118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.357132 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.357282 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.357289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.358727 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.358731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.360488 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.360507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.363696 (+ 3189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.363699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.363807 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.363833 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.363923 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.364008 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":218,"thread_start_us":137,"threads_started":1}
I20260925 16:27:03.364279 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.356175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.0.98.1:33793)
0925 16:27:03.356467 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.356474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.356507 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:27:03.356588 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.356592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.356798 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.356960 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.357393 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.358609 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.361039 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.363546 (+ 2507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.363676 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.363827 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.363975 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.364112 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260925 16:27:03.367108 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 64bea8a12b9a451587e7f69cbf56ee70 (DEFAULT_TABLE table=test-table [id=afc73bb40fe84308828452923fec969c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:03.367437 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 64bea8a12b9a451587e7f69cbf56ee70 (DEFAULT_TABLE table=test-table [id=afc73bb40fe84308828452923fec969c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:03.367861 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 64bea8a12b9a451587e7f69cbf56ee70 (DEFAULT_TABLE table=test-table [id=afc73bb40fe84308828452923fec969c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:03.368690 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bea8a12b9a451587e7f69cbf56ee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.369072 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bea8a12b9a451587e7f69cbf56ee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.369063 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bea8a12b9a451587e7f69cbf56ee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.380584 2630 tablet_bootstrap.cc:492] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Bootstrap starting.
I20260925 16:27:03.380584 2631 tablet_bootstrap.cc:492] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Bootstrap starting.
I20260925 16:27:03.382303 2629 tablet_bootstrap.cc:492] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Bootstrap starting.
I20260925 16:27:03.383491 2630 tablet_bootstrap.cc:654] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.383491 2631 tablet_bootstrap.cc:654] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.384522 2631 log.cc:824] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:03.384517 2630 log.cc:824] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:03.385353 2629 tablet_bootstrap.cc:654] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.386535 2629 log.cc:824] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:03.387483 2630 tablet_bootstrap.cc:492] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: No bootstrap required, opened a new log
I20260925 16:27:03.387660 2631 tablet_bootstrap.cc:492] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: No bootstrap required, opened a new log
I20260925 16:27:03.387768 2630 ts_tablet_manager.cc:1397] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:27:03.387912 2631 ts_tablet_manager.cc:1397] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 16:27:03.388887 2629 tablet_bootstrap.cc:492] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: No bootstrap required, opened a new log
I20260925 16:27:03.389093 2629 ts_tablet_manager.cc:1397] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 16:27:03.391472 2630 raft_consensus.cc:348] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.391851 2630 raft_consensus.cc:374] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.391906 2630 raft_consensus.cc:729] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae69f4e3351498ba7cc458f3e8e5174, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.392486 2630 consensus_queue.cc:260] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.392627 2631 raft_consensus.cc:348] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.393023 2631 raft_consensus.cc:374] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.393087 2631 raft_consensus.cc:729] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.393810 2630 ts_tablet_manager.cc:1428] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 16:27:03.393723 2631 consensus_queue.cc:260] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.394239 2606 heartbeater.cc:503] Master 127.0.98.62:36155 was elected leader, sending a full tablet report...
I20260925 16:27:03.394265 2629 raft_consensus.cc:348] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.394573 2629 raft_consensus.cc:374] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.394631 2629 raft_consensus.cc:729] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.395076 2631 ts_tablet_manager.cc:1428] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260925 16:27:03.395175 2629 consensus_queue.cc:260] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.395565 2466 heartbeater.cc:503] Master 127.0.98.62:36155 was elected leader, sending a full tablet report...
I20260925 16:27:03.396327 2326 heartbeater.cc:503] Master 127.0.98.62:36155 was elected leader, sending a full tablet report...
I20260925 16:27:03.396440 2629 ts_tablet_manager.cc:1428] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260925 16:27:03.403986 2636 raft_consensus.cc:482] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:03.404297 2636 raft_consensus.cc:504] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.405938 2636 leader_election.cc:288] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419), cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793)
I20260925 16:27:03.411105 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.406515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36419 (local address 127.0.98.2:42231)
0925 16:27:03.406698 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.406718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.406849 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.407138 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.407142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.407159 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.407294 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.407300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.408271 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.408274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.409468 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.409481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.410544 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.410548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.410680 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.410704 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.410783 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.410905 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 16:27:03.411090 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.406619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42231 (local address 127.0.98.3:36419)
0925 16:27:03.406777 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.406782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.406801 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:27:03.406989 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.406992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.407047 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.407118 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.407427 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.408130 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.409624 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.410253 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.410365 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.410467 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.410815 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.410915 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:27:03.411558 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.407283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32915 (local address 127.0.98.1:33793)
0925 16:27:03.407464 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.407469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.407485 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:27:03.407540 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.407543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.407595 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.407677 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.408140 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.408669 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.410370 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.410859 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.410967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.411040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.411262 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.411401 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 16:27:03.412065 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bea8a12b9a451587e7f69cbf56ee70" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174" is_pre_election: true
I20260925 16:27:03.412388 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.406862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.98.2:32915)
0925 16:27:03.407241 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.407258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.407386 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.407834 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.407836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.407857 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.408004 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.408010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.408817 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.408821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.410218 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.410228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.410990 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.410994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.411108 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.411133 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.411808 (+ 675us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.411907 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260925 16:27:03.412532 2560 raft_consensus.cc:2454] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 0.
I20260925 16:27:03.413122 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bea8a12b9a451587e7f69cbf56ee70" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" is_pre_election: true
I20260925 16:27:03.413156 2354 leader_election.cc:302] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:03.413439 2280 raft_consensus.cc:2454] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 0.
I20260925 16:27:03.413543 2636 raft_consensus.cc:2781] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:03.413677 2636 raft_consensus.cc:482] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:03.413734 2636 raft_consensus.cc:3037] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.415740 2636 raft_consensus.cc:504] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.416194 2636 leader_election.cc:288] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 election: Requested vote from peers fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419), cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793)
I20260925 16:27:03.416807 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bea8a12b9a451587e7f69cbf56ee70" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6"
I20260925 16:27:03.416805 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bea8a12b9a451587e7f69cbf56ee70" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174"
I20260925 16:27:03.416972 2560 raft_consensus.cc:3037] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.416987 2280 raft_consensus.cc:3037] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.418669 2280 raft_consensus.cc:2454] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 1.
I20260925 16:27:03.418669 2560 raft_consensus.cc:2454] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 1.
I20260925 16:27:03.419183 2354 leader_election.cc:302] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, cd9f0f0ecfeb458da9c672a3a428a7b6; no voters:
I20260925 16:27:03.419440 2636 raft_consensus.cc:2781] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:03.419801 2636 raft_consensus.cc:686] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 LEADER]: Becoming Leader. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Running, Role: LEADER
I20260925 16:27:03.420270 2636 consensus_queue.cc:237] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } }
I20260925 16:27:03.424707 2119 catalog_manager.cc:5495] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } health_report { overall_health: HEALTHY } } }
Sep 25 16:27:03 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:03.479894 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.468345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.0.1:47846)
0925 16:27:03.468760 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.468779 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.468928 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.469281 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.469283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.469896 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:03.470029 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.470036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.470929 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.470931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.471570 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.471578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.472185 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.472188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.472269 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.473256 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:03.473278 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:03.475432 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:03.477527 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.477546 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.477554 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.478056 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.478477 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:03.478480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:03.478482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:03.478592 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:03.479103 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:03.479112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:03.479304 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.479426 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.479645 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":191,"threads_started":1}
I20260925 16:27:03.480132 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.468491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.0.98.62:36155)
0925 16:27:03.468847 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.468852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.468900 (+ 48us) server_negotiation.cc:407] Connection header received
0925 16:27:03.469043 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.469047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.469114 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.469244 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:03.470205 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.470779 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.471781 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.472021 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.472127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.475747 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:03.475752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:03.475755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:03.475796 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:03.477330 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.478221 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.478226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.478227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.478336 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:03.478742 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:03.478745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:03.478746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:03.478968 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:03.479066 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.479580 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.479848 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 16:27:03.486303 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:27:03.487315 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 16:27:03.489183 2327 tablet.cc:2369] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:03.494298 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 02b17a6a7d094d37a9dcee56ac5de898 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a48ab87ceee548eababc3ece5d731b3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:03.494298 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 02b17a6a7d094d37a9dcee56ac5de898 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a48ab87ceee548eababc3ece5d731b3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:03.494431 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 02b17a6a7d094d37a9dcee56ac5de898 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a48ab87ceee548eababc3ece5d731b3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:03.495380 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b17a6a7d094d37a9dcee56ac5de898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.495380 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b17a6a7d094d37a9dcee56ac5de898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.495384 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b17a6a7d094d37a9dcee56ac5de898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.498656 2630 tablet_bootstrap.cc:492] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: Bootstrap starting.
I20260925 16:27:03.499118 2629 tablet_bootstrap.cc:492] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: Bootstrap starting.
I20260925 16:27:03.499773 2630 tablet_bootstrap.cc:654] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.500752 2629 tablet_bootstrap.cc:654] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.501106 2630 tablet_bootstrap.cc:492] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: No bootstrap required, opened a new log
I20260925 16:27:03.501199 2630 ts_tablet_manager.cc:1397] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:03.501674 2630 raft_consensus.cc:348] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.501845 2630 raft_consensus.cc:374] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.501885 2630 raft_consensus.cc:729] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae69f4e3351498ba7cc458f3e8e5174, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.501873 2631 tablet_bootstrap.cc:492] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Bootstrap starting.
I20260925 16:27:03.502082 2630 consensus_queue.cc:260] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.502396 2630 ts_tablet_manager.cc:1428] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:03.502517 2635 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.502772 2635 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:03.503294 2631 tablet_bootstrap.cc:654] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.503669 2629 tablet_bootstrap.cc:492] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: No bootstrap required, opened a new log
I20260925 16:27:03.503800 2629 ts_tablet_manager.cc:1397] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260925 16:27:03.504478 2629 raft_consensus.cc:348] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.504639 2629 raft_consensus.cc:374] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.504676 2629 raft_consensus.cc:729] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.504860 2629 consensus_queue.cc:260] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.505089 2631 tablet_bootstrap.cc:492] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: No bootstrap required, opened a new log
I20260925 16:27:03.505206 2631 ts_tablet_manager.cc:1397] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:03.505203 2637 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.505378 2637 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:03.505793 2631 raft_consensus.cc:348] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.505890 2629 ts_tablet_manager.cc:1428] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 16:27:03.505925 2631 raft_consensus.cc:374] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.505973 2631 raft_consensus.cc:729] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.506122 2631 consensus_queue.cc:260] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.506441 2636 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.506572 2631 ts_tablet_manager.cc:1428] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:03.506621 2636 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 16:27:03.511175 2467 tablet.cc:2369] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:27:03.527799 2607 tablet.cc:2369] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:03.553114 2637 raft_consensus.cc:482] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:03.553331 2637 raft_consensus.cc:504] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.554847 2637 leader_election.cc:288] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067), fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:03.559526 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.555252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36419 (local address 127.0.98.1:49935)
0925 16:27:03.555761 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.555773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.555947 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.556267 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.556270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.556288 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.556463 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.556471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.557229 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.557233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.558266 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.558278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.558994 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.558998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.559108 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.559133 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.559206 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.559311 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":63,"spinlock_wait_cycles":17408,"thread_start_us":160,"threads_started":1}
I20260925 16:27:03.559494 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.555437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49935 (local address 127.0.98.3:36419)
0925 16:27:03.555564 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.555569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.556058 (+ 489us) server_negotiation.cc:407] Connection header received
0925 16:27:03.556122 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.556124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.556178 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.556242 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.556565 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.557116 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.558406 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.558846 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.558946 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.559058 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.559230 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.559319 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:27:03.560375 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b17a6a7d094d37a9dcee56ac5de898" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174" is_pre_election: true
I20260925 16:27:03.560602 2560 raft_consensus.cc:2454] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 0.
I20260925 16:27:03.560705 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.555325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41863 (local address 127.0.98.2:33067)
0925 16:27:03.555501 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.555506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.555999 (+ 493us) server_negotiation.cc:407] Connection header received
0925 16:27:03.556101 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.556104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.556167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.556291 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.557413 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.558169 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.559421 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.559997 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.560130 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.560205 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.560422 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.560552 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 16:27:03.561097 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.555252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33067 (local address 127.0.98.1:41863)
0925 16:27:03.555755 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.555773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.555947 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.556374 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.556378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.556413 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.557153 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.557166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.558369 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.558374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.559257 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.559270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.560147 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.560150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.560264 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.560290 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.560383 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.560486 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"spinlock_wait_cycles":31360,"thread_start_us":202,"threads_started":1}
I20260925 16:27:03.561165 2214 leader_election.cc:302] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd9f0f0ecfeb458da9c672a3a428a7b6, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:03.561813 2637 raft_consensus.cc:2781] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:03.561901 2637 raft_consensus.cc:482] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:03.561941 2637 raft_consensus.cc:3037] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.561947 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b17a6a7d094d37a9dcee56ac5de898" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" is_pre_election: true
I20260925 16:27:03.562286 2420 raft_consensus.cc:2454] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 0.
I20260925 16:27:03.563213 2637 raft_consensus.cc:504] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.563707 2637 leader_election.cc:288] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067), fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:03.564220 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b17a6a7d094d37a9dcee56ac5de898" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a"
I20260925 16:27:03.564378 2420 raft_consensus.cc:3037] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.564441 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b17a6a7d094d37a9dcee56ac5de898" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174"
I20260925 16:27:03.564581 2560 raft_consensus.cc:3037] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.565904 2560 raft_consensus.cc:2454] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 1.
I20260925 16:27:03.565904 2420 raft_consensus.cc:2454] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 1.
I20260925 16:27:03.566349 2211 leader_election.cc:302] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, cd9f0f0ecfeb458da9c672a3a428a7b6, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:03.566643 2637 raft_consensus.cc:2781] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:03.566939 2637 raft_consensus.cc:686] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 LEADER]: Becoming Leader. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Running, Role: LEADER
I20260925 16:27:03.567458 2637 consensus_queue.cc:237] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.570703 2652 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: TxnStatusTablet state changed. Reason: New leader cd9f0f0ecfeb458da9c672a3a428a7b6. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.570901 2652 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:03.571275 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:27:03.571560 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:27:03.572129 2122 catalog_manager.cc:5495] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:03.589565 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.585735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.98.1:33793)
0925 16:27:03.585901 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.585906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.585925 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:27:03.586178 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.586181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.586239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.586319 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.586578 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.587104 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.587985 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.588201 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.588287 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.588590 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.589092 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.589194 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.589271 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.589328 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 16:27:03.589573 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.585594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.0.1:58824)
0925 16:27:03.585842 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.585860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.586013 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.586357 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.586360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.586372 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:03.586477 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.586482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.587206 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.587210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.587863 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.587873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.588305 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.588309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.588392 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.588446 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.589182 (+ 736us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.589196 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.589261 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.589368 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 16:27:03.595566 2560 raft_consensus.cc:1261] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Refusing update from remote peer cd9f0f0ecfeb458da9c672a3a428a7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:03.595563 2420 raft_consensus.cc:1261] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Refusing update from remote peer cd9f0f0ecfeb458da9c672a3a428a7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:03.596334 2637 consensus_queue.cc:1035] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:03.596563 2652 consensus_queue.cc:1035] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:03.600114 2636 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: TxnStatusTablet state changed. Reason: New leader cd9f0f0ecfeb458da9c672a3a428a7b6. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.600293 2636 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:03.602993 2635 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: TxnStatusTablet state changed. Reason: New leader cd9f0f0ecfeb458da9c672a3a428a7b6. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.603219 2635 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:03.607232 2637 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.607470 2637 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:03.608702 2660 mvcc.cc:205] Tried to move back new op lower bound from 7333288442239197184 to 7333288442137161728. Current Snapshot: MvccSnapshot[applied={T|T < 7333288442239197184}]
I20260925 16:27:03.608899 2635 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.609055 2635 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P fae69f4e3351498ba7cc458f3e8e5174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:03.614163 2652 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.614356 2652 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P cd9f0f0ecfeb458da9c672a3a428a7b6: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:03.615893 2636 tablet_replica.cc:439] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } } }
I20260925 16:27:03.616122 2636 tablet_replica.cc:442] T 02b17a6a7d094d37a9dcee56ac5de898 P 8ac2d397f5404ea9aea1c997fb2cdf1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:03.623901 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.619102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.0.98.2:33067)
0925 16:27:03.619213 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.619218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.619232 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:27:03.619309 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.619312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.619375 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.619498 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.619833 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.620691 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.622173 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.622435 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.622545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.622872 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.623446 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.623559 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.623662 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.623735 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 16:27:03.639812 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.634557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.98.2:38193)
0925 16:27:03.635109 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.635131 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.635305 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.635747 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.635750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.635758 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:03.635932 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.635938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.636959 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.636962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.638085 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.638100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.638663 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.638666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.638777 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.638858 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.639376 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.639383 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.639455 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.639579 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260925 16:27:03.639786 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.635053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38193 (local address 127.0.98.1:33793)
0925 16:27:03.635476 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.635480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.635498 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:27:03.635566 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.635568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.635620 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.635707 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.636135 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.636806 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.638266 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.638504 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.638609 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.639146 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.639256 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.639355 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.639549 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.639621 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260925 16:27:03.650454 2560 raft_consensus.cc:1261] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Refusing update from remote peer 8ac2d397f5404ea9aea1c997fb2cdf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:03.650491 2280 raft_consensus.cc:1261] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Refusing update from remote peer 8ac2d397f5404ea9aea1c997fb2cdf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:03.651137 2636 consensus_queue.cc:1035] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:03.651379 2639 consensus_queue.cc:1035] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:03.656229 2658 mvcc.cc:205] Tried to move back new op lower bound from 7333288442466250752 to 7333288441533218816. Current Snapshot: MvccSnapshot[applied={T|T < 7333288442466250752}]
I20260925 16:27:03.683144 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.678763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45633 (local address 127.0.98.2:33067)
0925 16:27:03.678967 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.678972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.678988 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:27:03.679173 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.679176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.679234 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.679298 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.679613 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.680519 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.681744 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.682098 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.682174 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.682623 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.682711 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.682776 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.682905 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.682973 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 16:27:03.683606 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.678522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33067 (local address 127.0.98.1:45633)
0925 16:27:03.678908 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.678925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.679052 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.679322 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.679325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.679332 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:03.679456 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.679463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.680703 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.680707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.681552 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.681563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.682300 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.682304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.682385 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.682449 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.682800 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.682807 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.683320 (+ 513us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.683441 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":119,"threads_started":1}
I20260925 16:27:03.729841 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:03.730197 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:03.734719 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Sending DeleteTablet for 3 replicas of tablet 64bea8a12b9a451587e7f69cbf56ee70
I20260925 16:27:03.736352 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bea8a12b9a451587e7f69cbf56ee70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260925 16:27:03.738360 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bea8a12b9a451587e7f69cbf56ee70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56490
I20260925 16:27:03.738808 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:27:03.739353 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bea8a12b9a451587e7f69cbf56ee70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58822
W20260925 16:27:03.739521 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:03.750402 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3e154269fa4d16b69d41ef5fc3aca3 (DEFAULT_TABLE table=test-table [id=c00f74db002746dc83f74d0e5d9a9471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:03.750926 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3e154269fa4d16b69d41ef5fc3aca3 (DEFAULT_TABLE table=test-table [id=c00f74db002746dc83f74d0e5d9a9471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:03.751107 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3e154269fa4d16b69d41ef5fc3aca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.751309 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3e154269fa4d16b69d41ef5fc3aca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.754199 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3e154269fa4d16b69d41ef5fc3aca3 (DEFAULT_TABLE table=test-table [id=c00f74db002746dc83f74d0e5d9a9471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:03.754583 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3e154269fa4d16b69d41ef5fc3aca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:03.754694 2631 tablet_bootstrap.cc:492] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Bootstrap starting.
I20260925 16:27:03.755832 2631 tablet_bootstrap.cc:654] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.757661 2630 tablet_bootstrap.cc:492] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Bootstrap starting.
I20260925 16:27:03.758715 2630 tablet_bootstrap.cc:654] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.759346 2629 tablet_bootstrap.cc:492] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Bootstrap starting.
I20260925 16:27:03.760982 2629 tablet_bootstrap.cc:654] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:03.761119 2630 tablet_bootstrap.cc:492] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: No bootstrap required, opened a new log
I20260925 16:27:03.761228 2630 ts_tablet_manager.cc:1397] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 16:27:03.762728 2630 raft_consensus.cc:348] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.762948 2630 raft_consensus.cc:374] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.762990 2630 raft_consensus.cc:729] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae69f4e3351498ba7cc458f3e8e5174, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.763164 2630 consensus_queue.cc:260] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.763525 2630 ts_tablet_manager.cc:1428] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:03.766189 2679 tablet_replica.cc:330] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: stopping tablet replica
I20260925 16:27:03.766498 2678 tablet_replica.cc:330] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: stopping tablet replica
I20260925 16:27:03.767119 2678 raft_consensus.cc:2229] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:03.767572 2679 raft_consensus.cc:2229] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:03.768352 2629 tablet_bootstrap.cc:492] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: No bootstrap required, opened a new log
I20260925 16:27:03.770009 2629 ts_tablet_manager.cc:1397] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260925 16:27:03.770617 2629 raft_consensus.cc:348] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.770773 2629 raft_consensus.cc:374] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.770815 2629 raft_consensus.cc:729] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.770979 2629 consensus_queue.cc:260] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.771288 2629 ts_tablet_manager.cc:1428] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:27:03.771628 2679 raft_consensus.cc:2258] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:03.772042 2678 raft_consensus.cc:2258] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:03.772511 2677 tablet_replica.cc:330] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: stopping tablet replica
I20260925 16:27:03.773356 2677 raft_consensus.cc:2229] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:03.773818 2677 raft_consensus.cc:2258] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:03.774816 2631 tablet_bootstrap.cc:492] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: No bootstrap required, opened a new log
I20260925 16:27:03.774930 2631 ts_tablet_manager.cc:1397] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20260925 16:27:03.775429 2631 raft_consensus.cc:348] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.775574 2631 raft_consensus.cc:374] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:03.775621 2631 raft_consensus.cc:729] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Initialized, Role: FOLLOWER
I20260925 16:27:03.775763 2631 consensus_queue.cc:260] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.776031 2631 ts_tablet_manager.cc:1428] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:03.777644 2678 ts_tablet_manager.cc:1905] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:03.778764 2677 ts_tablet_manager.cc:1905] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:03.782904 2678 ts_tablet_manager.cc:1918] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:03.783064 2678 log.cc:1196] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/64bea8a12b9a451587e7f69cbf56ee70
I20260925 16:27:03.783265 2679 ts_tablet_manager.cc:1905] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:03.783880 2678 ts_tablet_manager.cc:1939] T 64bea8a12b9a451587e7f69cbf56ee70 P fae69f4e3351498ba7cc458f3e8e5174: Deleting consensus metadata
I20260925 16:27:03.785918 2110 catalog_manager.cc:4841] TS fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419): tablet 64bea8a12b9a451587e7f69cbf56ee70 (table test-table [id=afc73bb40fe84308828452923fec969c]) successfully deleted
I20260925 16:27:03.787637 2677 ts_tablet_manager.cc:1918] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:03.787829 2677 log.cc:1196] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/64bea8a12b9a451587e7f69cbf56ee70
I20260925 16:27:03.788203 2677 ts_tablet_manager.cc:1939] T 64bea8a12b9a451587e7f69cbf56ee70 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting consensus metadata
I20260925 16:27:03.788753 2679 ts_tablet_manager.cc:1918] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:03.788889 2679 log.cc:1196] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/64bea8a12b9a451587e7f69cbf56ee70
I20260925 16:27:03.789187 2679 ts_tablet_manager.cc:1939] T 64bea8a12b9a451587e7f69cbf56ee70 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting consensus metadata
I20260925 16:27:03.790074 2107 catalog_manager.cc:4841] TS 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067): tablet 64bea8a12b9a451587e7f69cbf56ee70 (table test-table [id=afc73bb40fe84308828452923fec969c]) successfully deleted
I20260925 16:27:03.790606 2110 catalog_manager.cc:4841] TS cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793): tablet 64bea8a12b9a451587e7f69cbf56ee70 (table test-table [id=afc73bb40fe84308828452923fec969c]) successfully deleted
I20260925 16:27:03.838335 2635 raft_consensus.cc:482] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:03.838577 2635 raft_consensus.cc:504] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.840253 2635 leader_election.cc:288] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793), 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067)
I20260925 16:27:03.846318 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.841083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49369 (local address 127.0.98.2:33067)
0925 16:27:03.841215 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.841219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.841456 (+ 237us) server_negotiation.cc:407] Connection header received
0925 16:27:03.841600 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.841603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.841653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.841737 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.842142 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.842901 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.844790 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.845466 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.845600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.845709 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.846051 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.846173 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:27:03.846378 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.840566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33067 (local address 127.0.98.3:49369)
0925 16:27:03.841372 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.841385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.841472 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.841843 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.841847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.841872 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.842012 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.842018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.843033 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.843037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.844606 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.844621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.845681 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.845686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.845858 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.845911 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.846012 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.846135 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":345,"thread_start_us":44,"threads_started":1}
I20260925 16:27:03.846985 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3e154269fa4d16b69d41ef5fc3aca3" candidate_uuid: "fae69f4e3351498ba7cc458f3e8e5174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" is_pre_election: true
I20260925 16:27:03.847205 2420 raft_consensus.cc:2454] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fae69f4e3351498ba7cc458f3e8e5174 in term 0.
I20260925 16:27:03.847384 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.840857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49557 (local address 127.0.98.1:33793)
0925 16:27:03.841043 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.841049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.841146 (+ 97us) server_negotiation.cc:407] Connection header received
0925 16:27:03.841409 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.841412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.841513 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.841644 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:03.842029 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.842867 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.845852 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.846532 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.846668 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.846747 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.847092 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.847248 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:27:03.847674 2491 leader_election.cc:302] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:03.847805 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.840566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.98.3:49557)
0925 16:27:03.841030 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.841048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.841188 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.841652 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.841655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.841668 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:03.841847 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.841856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.843033 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.843037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.844606 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.844621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.846673 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.846677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.846830 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.846857 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.847528 (+ 671us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.847636 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":44,"thread_start_us":119,"threads_started":1}
I20260925 16:27:03.848003 2635 raft_consensus.cc:2781] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:03.848070 2635 raft_consensus.cc:482] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:03.848127 2635 raft_consensus.cc:3037] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.848664 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3e154269fa4d16b69d41ef5fc3aca3" candidate_uuid: "fae69f4e3351498ba7cc458f3e8e5174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" is_pre_election: true
I20260925 16:27:03.848956 2280 raft_consensus.cc:2454] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fae69f4e3351498ba7cc458f3e8e5174 in term 0.
I20260925 16:27:03.850140 2635 raft_consensus.cc:504] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.850649 2635 leader_election.cc:288] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [CANDIDATE]: Term 1 election: Requested vote from peers cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793), 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067)
I20260925 16:27:03.851150 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3e154269fa4d16b69d41ef5fc3aca3" candidate_uuid: "fae69f4e3351498ba7cc458f3e8e5174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6"
I20260925 16:27:03.851297 2280 raft_consensus.cc:3037] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.851320 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3e154269fa4d16b69d41ef5fc3aca3" candidate_uuid: "fae69f4e3351498ba7cc458f3e8e5174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a"
I20260925 16:27:03.851539 2420 raft_consensus.cc:3037] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:03.852375 2280 raft_consensus.cc:2454] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fae69f4e3351498ba7cc458f3e8e5174 in term 1.
I20260925 16:27:03.852907 2494 leader_election.cc:302] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd9f0f0ecfeb458da9c672a3a428a7b6, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:03.853071 2420 raft_consensus.cc:2454] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fae69f4e3351498ba7cc458f3e8e5174 in term 1.
I20260925 16:27:03.853124 2635 raft_consensus.cc:2781] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:03.853312 2635 raft_consensus.cc:686] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 LEADER]: Becoming Leader. State: Replica: fae69f4e3351498ba7cc458f3e8e5174, State: Running, Role: LEADER
I20260925 16:27:03.853621 2635 consensus_queue.cc:237] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:03.857276 2122 catalog_manager.cc:5495] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 reported cstate change: term changed from 0 to 1, leader changed from <none> to fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fae69f4e3351498ba7cc458f3e8e5174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } health_report { overall_health: HEALTHY } } }
I20260925 16:27:03.883216 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.879408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.0.98.3:36419)
0925 16:27:03.879594 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.879598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.879612 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:27:03.879700 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.879702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.879766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.879867 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.880117 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.880724 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.881581 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.881928 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.882021 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.882311 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.882821 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.882927 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.883006 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.883066 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 16:27:03.897723 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.892767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.98.3:52167)
0925 16:27:03.893194 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.893209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.893349 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.893808 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.893813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.893830 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:03.893998 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.894006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.895190 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.895193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.896007 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.896023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.896728 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.896731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.896824 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.896891 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.897375 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.897382 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.897449 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.897540 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":175,"threads_started":1}
I20260925 16:27:03.897809 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.892888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52167 (local address 127.0.98.1:33793)
0925 16:27:03.893065 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.893070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.893307 (+ 237us) server_negotiation.cc:407] Connection header received
0925 16:27:03.893460 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.893464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.893535 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.893644 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.894171 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.894998 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.896190 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.896552 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.896694 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.897065 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.897210 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.897302 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.897486 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.897571 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:27:03.915957 2280 raft_consensus.cc:1261] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Refusing update from remote peer fae69f4e3351498ba7cc458f3e8e5174: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:03.916675 2420 raft_consensus.cc:1261] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Refusing update from remote peer fae69f4e3351498ba7cc458f3e8e5174: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:03.916551 2635 consensus_queue.cc:1035] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:03.917194 2685 consensus_queue.cc:1035] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:03.924844 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333288443554586624 to 7333288443308519424. Current Snapshot: MvccSnapshot[applied={T|T < 7333288443554586624}]
I20260925 16:27:03.939767 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.935241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36419 (local address 127.0.98.1:45251)
0925 16:27:03.935510 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:03.935525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:03.935611 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:03.936126 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:03.936129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:03.936137 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:03.936246 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.936251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.937148 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.937151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.938209 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:03.938220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.938866 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.938869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:03.938971 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.939016 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.939415 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.939422 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:03.939498 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:27:03.939599 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 16:27:03.940402 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:03.935744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45251 (local address 127.0.98.3:36419)
0925 16:27:03.935850 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:03.935854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:03.935866 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:27:03.935909 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:03.935911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:03.935959 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:03.936018 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:03.936394 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.937015 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.938402 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:03.938683 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:03.938844 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:03.939180 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:03.939301 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:03.939389 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:03.940182 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 16:27:03.940274 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 16:27:03.941159 2660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 16:27:05.530437 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:05.530717 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:05.533831 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Sending DeleteTablet for 3 replicas of tablet 3f3e154269fa4d16b69d41ef5fc3aca3
I20260925 16:27:05.534736 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3e154269fa4d16b69d41ef5fc3aca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58822
I20260925 16:27:05.534781 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3e154269fa4d16b69d41ef5fc3aca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260925 16:27:05.534812 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3e154269fa4d16b69d41ef5fc3aca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56490
I20260925 16:27:05.535813 2717 tablet_replica.cc:330] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: stopping tablet replica
I20260925 16:27:05.536031 2717 raft_consensus.cc:2229] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:05.536211 2717 raft_consensus.cc:2258] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:05.537007 2717 ts_tablet_manager.cc:1905] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:05.537600 2718 tablet_replica.cc:330] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: stopping tablet replica
I20260925 16:27:05.537788 2718 raft_consensus.cc:2229] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:05.537937 2718 raft_consensus.cc:2258] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:05.538125 2716 tablet_replica.cc:330] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: stopping tablet replica
I20260925 16:27:05.538282 2716 raft_consensus.cc:2229] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:05.538650 2716 raft_consensus.cc:2258] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:05.539215 2718 ts_tablet_manager.cc:1905] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:05.539357 2716 ts_tablet_manager.cc:1905] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:05.541093 2717 ts_tablet_manager.cc:1918] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:27:05.541261 2717 log.cc:1196] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/3f3e154269fa4d16b69d41ef5fc3aca3
I20260925 16:27:05.541527 2717 ts_tablet_manager.cc:1939] T 3f3e154269fa4d16b69d41ef5fc3aca3 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting consensus metadata
I20260925 16:27:05.542488 2716 ts_tablet_manager.cc:1918] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:27:05.542584 2716 log.cc:1196] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/3f3e154269fa4d16b69d41ef5fc3aca3
I20260925 16:27:05.542562 2110 catalog_manager.cc:4841] TS cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793): tablet 3f3e154269fa4d16b69d41ef5fc3aca3 (table test-table [id=c00f74db002746dc83f74d0e5d9a9471]) successfully deleted
I20260925 16:27:05.542871 2716 ts_tablet_manager.cc:1939] T 3f3e154269fa4d16b69d41ef5fc3aca3 P fae69f4e3351498ba7cc458f3e8e5174: Deleting consensus metadata
I20260925 16:27:05.543664 2110 catalog_manager.cc:4841] TS fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419): tablet 3f3e154269fa4d16b69d41ef5fc3aca3 (table test-table [id=c00f74db002746dc83f74d0e5d9a9471]) successfully deleted
I20260925 16:27:05.544241 2718 ts_tablet_manager.cc:1918] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:27:05.544361 2718 log.cc:1196] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/3f3e154269fa4d16b69d41ef5fc3aca3
I20260925 16:27:05.544631 2718 ts_tablet_manager.cc:1939] T 3f3e154269fa4d16b69d41ef5fc3aca3 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting consensus metadata
I20260925 16:27:05.545624 2107 catalog_manager.cc:4841] TS 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067): tablet 3f3e154269fa4d16b69d41ef5fc3aca3 (table test-table [id=c00f74db002746dc83f74d0e5d9a9471]) successfully deleted
Sep 25 16:27:05 dist-test-slave-31kt krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:27:05 dist-test-slave-31kt krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:05.571545 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:05.560093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.98.62:36155)
0925 16:27:05.560435 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:05.560443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:05.560488 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:27:05.560747 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:05.560751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:05.560818 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:05.560937 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:05.561909 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.562617 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.563715 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.564018 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.564141 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:05.567353 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:05.567359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:05.567361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:05.567412 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:05.569200 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:05.569923 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:05.569928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:05.569930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:05.570050 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:05.570499 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:05.570503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:05.570504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:05.570754 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:05.570894 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:05.571169 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 16:27:05.571315 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260925 16:27:05.574965 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:05.575678 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:05.582361 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 93afe2c4de234ad393c7463d7f5c7f89 (DEFAULT_TABLE table=test-table [id=978b8c2c53a24503ae763c20bee7900f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:05.582563 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 93afe2c4de234ad393c7463d7f5c7f89 (DEFAULT_TABLE table=test-table [id=978b8c2c53a24503ae763c20bee7900f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:05.582723 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93afe2c4de234ad393c7463d7f5c7f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:05.582662 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 93afe2c4de234ad393c7463d7f5c7f89 (DEFAULT_TABLE table=test-table [id=978b8c2c53a24503ae763c20bee7900f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:05.582921 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93afe2c4de234ad393c7463d7f5c7f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:05.582954 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93afe2c4de234ad393c7463d7f5c7f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:05.586033 2727 tablet_bootstrap.cc:492] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Bootstrap starting.
I20260925 16:27:05.586920 2729 tablet_bootstrap.cc:492] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Bootstrap starting.
I20260925 16:27:05.587037 2727 tablet_bootstrap.cc:654] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:05.587857 2729 tablet_bootstrap.cc:654] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:05.588574 2727 tablet_bootstrap.cc:492] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: No bootstrap required, opened a new log
I20260925 16:27:05.588685 2727 ts_tablet_manager.cc:1397] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:05.589152 2727 raft_consensus.cc:348] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.589291 2727 raft_consensus.cc:374] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:05.589336 2727 raft_consensus.cc:729] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Initialized, Role: FOLLOWER
I20260925 16:27:05.589396 2729 tablet_bootstrap.cc:492] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: No bootstrap required, opened a new log
I20260925 16:27:05.589483 2727 consensus_queue.cc:260] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.589594 2729 ts_tablet_manager.cc:1397] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:05.590112 2729 raft_consensus.cc:348] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.590231 2729 raft_consensus.cc:374] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:05.590253 2729 raft_consensus.cc:729] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae69f4e3351498ba7cc458f3e8e5174, State: Initialized, Role: FOLLOWER
I20260925 16:27:05.590392 2729 consensus_queue.cc:260] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.590541 2727 ts_tablet_manager.cc:1428] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:05.591224 2729 ts_tablet_manager.cc:1428] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:05.588815 2728 tablet_bootstrap.cc:492] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Bootstrap starting.
I20260925 16:27:05.592538 2728 tablet_bootstrap.cc:654] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:05.594444 2728 tablet_bootstrap.cc:492] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: No bootstrap required, opened a new log
I20260925 16:27:05.594551 2728 ts_tablet_manager.cc:1397] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 16:27:05.594983 2728 raft_consensus.cc:348] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.595098 2728 raft_consensus.cc:374] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:05.595126 2728 raft_consensus.cc:729] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Initialized, Role: FOLLOWER
I20260925 16:27:05.595261 2728 consensus_queue.cc:260] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.595548 2728 ts_tablet_manager.cc:1428] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:27:05.608935 2732 raft_consensus.cc:482] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:05.609241 2732 raft_consensus.cc:504] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.609823 2732 leader_election.cc:288] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793), fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:05.610445 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93afe2c4de234ad393c7463d7f5c7f89" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174" is_pre_election: true
I20260925 16:27:05.610445 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93afe2c4de234ad393c7463d7f5c7f89" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" is_pre_election: true
I20260925 16:27:05.610706 2280 raft_consensus.cc:2454] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 0.
I20260925 16:27:05.610706 2560 raft_consensus.cc:2454] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 0.
I20260925 16:27:05.611166 2354 leader_election.cc:302] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:05.611426 2732 raft_consensus.cc:2781] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:05.611508 2732 raft_consensus.cc:482] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:05.611536 2732 raft_consensus.cc:3037] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:05.612696 2732 raft_consensus.cc:504] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.613113 2732 leader_election.cc:288] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 election: Requested vote from peers cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793), fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:05.613868 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93afe2c4de234ad393c7463d7f5c7f89" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174"
I20260925 16:27:05.613867 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93afe2c4de234ad393c7463d7f5c7f89" candidate_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6"
I20260925 16:27:05.614086 2280 raft_consensus.cc:3037] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:05.614086 2560 raft_consensus.cc:3037] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:05.615622 2280 raft_consensus.cc:2454] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 1.
I20260925 16:27:05.615622 2560 raft_consensus.cc:2454] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac2d397f5404ea9aea1c997fb2cdf1a in term 1.
I20260925 16:27:05.616124 2354 leader_election.cc:302] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, cd9f0f0ecfeb458da9c672a3a428a7b6; no voters:
I20260925 16:27:05.616591 2732 raft_consensus.cc:2781] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:05.616880 2732 raft_consensus.cc:686] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 LEADER]: Becoming Leader. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Running, Role: LEADER
I20260925 16:27:05.617077 2732 consensus_queue.cc:237] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.619271 2122 catalog_manager.cc:5495] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:05.644455 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:05.639420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.0.98.2:33067)
0925 16:27:05.639847 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:05.639856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:05.639869 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:27:05.639936 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:05.639938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:05.639991 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:05.640081 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:05.640378 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.641191 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.642713 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.643114 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.643246 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:05.643717 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:05.643881 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:05.643974 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:05.644127 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 16:27:05.644218 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260925 16:27:05.653740 2280 raft_consensus.cc:1261] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Refusing update from remote peer 8ac2d397f5404ea9aea1c997fb2cdf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:05.653728 2560 raft_consensus.cc:1261] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Refusing update from remote peer 8ac2d397f5404ea9aea1c997fb2cdf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:05.654377 2734 consensus_queue.cc:1035] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:05.654613 2732 consensus_queue.cc:1035] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:05.698441 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:05.698671 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:05.701700 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Sending DeleteTablet for 3 replicas of tablet 93afe2c4de234ad393c7463d7f5c7f89
I20260925 16:27:05.702836 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 93afe2c4de234ad393c7463d7f5c7f89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56490
I20260925 16:27:05.702870 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 93afe2c4de234ad393c7463d7f5c7f89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260925 16:27:05.703109 2716 tablet_replica.cc:330] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: stopping tablet replica
I20260925 16:27:05.703169 2718 tablet_replica.cc:330] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: stopping tablet replica
I20260925 16:27:05.703127 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 93afe2c4de234ad393c7463d7f5c7f89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58822
I20260925 16:27:05.703365 2718 raft_consensus.cc:2229] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:05.703428 2716 raft_consensus.cc:2229] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:05.703476 2717 tablet_replica.cc:330] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: stopping tablet replica
I20260925 16:27:05.703644 2716 raft_consensus.cc:2258] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:05.703655 2718 raft_consensus.cc:2258] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:05.703706 2717 raft_consensus.cc:2229] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:05.703846 2717 raft_consensus.cc:2258] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:05.704380 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:27:05.704752 2718 ts_tablet_manager.cc:1905] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:05.704752 2716 ts_tablet_manager.cc:1905] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:27:05.705054 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:05.708819 2716 ts_tablet_manager.cc:1918] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:05.708819 2718 ts_tablet_manager.cc:1918] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:05.708961 2718 log.cc:1196] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/93afe2c4de234ad393c7463d7f5c7f89
I20260925 16:27:05.708961 2716 log.cc:1196] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/93afe2c4de234ad393c7463d7f5c7f89
I20260925 16:27:05.709242 2718 ts_tablet_manager.cc:1939] T 93afe2c4de234ad393c7463d7f5c7f89 P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting consensus metadata
I20260925 16:27:05.709242 2716 ts_tablet_manager.cc:1939] T 93afe2c4de234ad393c7463d7f5c7f89 P fae69f4e3351498ba7cc458f3e8e5174: Deleting consensus metadata
I20260925 16:27:05.710983 2717 ts_tablet_manager.cc:1905] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:05.710985 2107 catalog_manager.cc:4841] TS 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067): tablet 93afe2c4de234ad393c7463d7f5c7f89 (table test-table [id=978b8c2c53a24503ae763c20bee7900f]) successfully deleted
I20260925 16:27:05.710985 2110 catalog_manager.cc:4841] TS fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419): tablet 93afe2c4de234ad393c7463d7f5c7f89 (table test-table [id=978b8c2c53a24503ae763c20bee7900f]) successfully deleted
I20260925 16:27:05.712739 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 58ab2b7f53464929ac76b61a6d5b3f1c (DEFAULT_TABLE table=test-table [id=1d1676e65ee94e7494d38769e5270de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:05.712704 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 58ab2b7f53464929ac76b61a6d5b3f1c (DEFAULT_TABLE table=test-table [id=1d1676e65ee94e7494d38769e5270de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:05.712872 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 58ab2b7f53464929ac76b61a6d5b3f1c (DEFAULT_TABLE table=test-table [id=1d1676e65ee94e7494d38769e5270de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:05.713095 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ab2b7f53464929ac76b61a6d5b3f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:05.713096 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ab2b7f53464929ac76b61a6d5b3f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:05.713140 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ab2b7f53464929ac76b61a6d5b3f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:05.714864 2717 ts_tablet_manager.cc:1918] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:05.714960 2717 log.cc:1196] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/93afe2c4de234ad393c7463d7f5c7f89
I20260925 16:27:05.715260 2717 ts_tablet_manager.cc:1939] T 93afe2c4de234ad393c7463d7f5c7f89 P cd9f0f0ecfeb458da9c672a3a428a7b6: Deleting consensus metadata
I20260925 16:27:05.716144 2728 tablet_bootstrap.cc:492] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6: Bootstrap starting.
I20260925 16:27:05.716236 2727 tablet_bootstrap.cc:492] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: Bootstrap starting.
I20260925 16:27:05.717295 2728 tablet_bootstrap.cc:654] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:05.717295 2727 tablet_bootstrap.cc:654] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:05.717994 2729 tablet_bootstrap.cc:492] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Bootstrap starting.
I20260925 16:27:05.718667 2728 tablet_bootstrap.cc:492] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6: No bootstrap required, opened a new log
I20260925 16:27:05.718772 2728 ts_tablet_manager.cc:1397] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260925 16:27:05.718982 2729 tablet_bootstrap.cc:654] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:05.719128 2727 tablet_bootstrap.cc:492] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: No bootstrap required, opened a new log
I20260925 16:27:05.719226 2727 ts_tablet_manager.cc:1397] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:27:05.719316 2728 raft_consensus.cc:348] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.719460 2728 raft_consensus.cc:374] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:05.719492 2728 raft_consensus.cc:729] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Initialized, Role: FOLLOWER
I20260925 16:27:05.719648 2728 consensus_queue.cc:260] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.719686 2727 raft_consensus.cc:348] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.719791 2727 raft_consensus.cc:374] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:05.719821 2727 raft_consensus.cc:729] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d397f5404ea9aea1c997fb2cdf1a, State: Initialized, Role: FOLLOWER
I20260925 16:27:05.719945 2727 consensus_queue.cc:260] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.720034 2728 ts_tablet_manager.cc:1428] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:27:05.720178 2729 tablet_bootstrap.cc:492] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: No bootstrap required, opened a new log
I20260925 16:27:05.720221 2727 ts_tablet_manager.cc:1428] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:27:05.720276 2729 ts_tablet_manager.cc:1397] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:05.720705 2729 raft_consensus.cc:348] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.720835 2729 raft_consensus.cc:374] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:05.720873 2729 raft_consensus.cc:729] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae69f4e3351498ba7cc458f3e8e5174, State: Initialized, Role: FOLLOWER
I20260925 16:27:05.721002 2729 consensus_queue.cc:260] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.721326 2729 ts_tablet_manager.cc:1428] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:05.722427 2110 catalog_manager.cc:4841] TS cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793): tablet 93afe2c4de234ad393c7463d7f5c7f89 (table test-table [id=978b8c2c53a24503ae763c20bee7900f]) successfully deleted
I20260925 16:27:05.757987 2702 raft_consensus.cc:482] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:05.758167 2702 raft_consensus.cc:504] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.758704 2702 leader_election.cc:288] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067), fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:05.759552 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ab2b7f53464929ac76b61a6d5b3f1c" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" is_pre_election: true
I20260925 16:27:05.759573 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ab2b7f53464929ac76b61a6d5b3f1c" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174" is_pre_election: true
I20260925 16:27:05.759749 2419 raft_consensus.cc:2454] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 0.
I20260925 16:27:05.759828 2560 raft_consensus.cc:2454] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 0.
I20260925 16:27:05.760216 2211 leader_election.cc:302] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, cd9f0f0ecfeb458da9c672a3a428a7b6, fae69f4e3351498ba7cc458f3e8e5174; no voters:
I20260925 16:27:05.760489 2702 raft_consensus.cc:2781] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:05.760564 2702 raft_consensus.cc:482] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:05.760725 2702 raft_consensus.cc:3037] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:05.762512 2702 raft_consensus.cc:504] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.762987 2702 leader_election.cc:288] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac2d397f5404ea9aea1c997fb2cdf1a (127.0.98.2:33067), fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419)
I20260925 16:27:05.763636 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ab2b7f53464929ac76b61a6d5b3f1c" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a"
I20260925 16:27:05.763799 2419 raft_consensus.cc:3037] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:05.763760 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ab2b7f53464929ac76b61a6d5b3f1c" candidate_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae69f4e3351498ba7cc458f3e8e5174"
I20260925 16:27:05.763996 2560 raft_consensus.cc:3037] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:05.765137 2419 raft_consensus.cc:2454] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 1.
I20260925 16:27:05.765455 2560 raft_consensus.cc:2454] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9f0f0ecfeb458da9c672a3a428a7b6 in term 1.
I20260925 16:27:05.765611 2211 leader_election.cc:302] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac2d397f5404ea9aea1c997fb2cdf1a, cd9f0f0ecfeb458da9c672a3a428a7b6; no voters:
I20260925 16:27:05.766378 2702 raft_consensus.cc:2781] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:05.766518 2702 raft_consensus.cc:686] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [term 1 LEADER]: Becoming Leader. State: Replica: cd9f0f0ecfeb458da9c672a3a428a7b6, State: Running, Role: LEADER
I20260925 16:27:05.766705 2702 consensus_queue.cc:237] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } }
I20260925 16:27:05.768930 2119 catalog_manager.cc:5495] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd9f0f0ecfeb458da9c672a3a428a7b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:05.795065 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:05.790678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.0.98.1:33793)
0925 16:27:05.791087 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:05.791096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:05.791116 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:05.791183 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:05.791185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:05.791245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:05.791351 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:05.791787 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.792427 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.793545 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.793898 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.794015 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:05.794374 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:05.794495 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:05.794561 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:05.794740 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 16:27:05.794841 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":129,"threads_started":1}
I20260925 16:27:05.809929 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:05.805486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53481 (local address 127.0.98.1:33793)
0925 16:27:05.805674 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:05.805680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:05.805788 (+ 108us) server_negotiation.cc:407] Connection header received
0925 16:27:05.805999 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:05.806002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:05.806060 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:05.806151 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:05.806635 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.807257 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.808306 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.808608 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.808711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:05.809134 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:05.809258 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:05.809339 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:05.809595 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 16:27:05.809682 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 16:27:05.809957 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:05.805129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.98.1:53481)
0925 16:27:05.805658 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:05.805674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:05.805880 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:05.806227 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:05.806230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:05.806240 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:05.806474 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:05.806482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.807431 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.807434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:05.808150 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:05.808162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:05.808782 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:05.808785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:05.808906 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:05.808955 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:05.809391 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:05.809399 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:27:05.809470 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 16:27:05.809703 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":1,"thread_start_us":270,"threads_started":1}
I20260925 16:27:05.818356 2560 raft_consensus.cc:1261] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Refusing update from remote peer cd9f0f0ecfeb458da9c672a3a428a7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:05.818660 2419 raft_consensus.cc:1261] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Refusing update from remote peer cd9f0f0ecfeb458da9c672a3a428a7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:05.818898 2652 consensus_queue.cc:1035] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae69f4e3351498ba7cc458f3e8e5174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:05.819278 2652 consensus_queue.cc:1035] T 58ab2b7f53464929ac76b61a6d5b3f1c P cd9f0f0ecfeb458da9c672a3a428a7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d397f5404ea9aea1c997fb2cdf1a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:27:05.839421 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 16:27:07.422802 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:07.423038 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:07.426781 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 031e75bd4a8048dfa9b386fce17e6f78: Sending DeleteTablet for 3 replicas of tablet 58ab2b7f53464929ac76b61a6d5b3f1c
I20260925 16:27:07.427682 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ab2b7f53464929ac76b61a6d5b3f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56490
I20260925 16:27:07.427709 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ab2b7f53464929ac76b61a6d5b3f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58822
I20260925 16:27:07.427716 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ab2b7f53464929ac76b61a6d5b3f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260925 16:27:07.428085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 2195
I20260925 16:27:07.429346 2773 tablet_replica.cc:330] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: stopping tablet replica
I20260925 16:27:07.429566 2773 raft_consensus.cc:2229] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:07.429787 2773 raft_consensus.cc:2258] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:07.429982 2775 tablet_replica.cc:330] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: stopping tablet replica
I20260925 16:27:07.430141 2775 raft_consensus.cc:2229] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:07.430266 2775 raft_consensus.cc:2258] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:07.430751 2773 ts_tablet_manager.cc:1905] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:07.434562 2773 ts_tablet_manager.cc:1918] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 16:27:07.434669 2773 log.cc:1196] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/58ab2b7f53464929ac76b61a6d5b3f1c
I20260925 16:27:07.434927 2773 ts_tablet_manager.cc:1939] T 58ab2b7f53464929ac76b61a6d5b3f1c P fae69f4e3351498ba7cc458f3e8e5174: Deleting consensus metadata
I20260925 16:27:07.435480 2775 ts_tablet_manager.cc:1905] T 58ab2b7f53464929ac76b61a6d5b3f1c P 8ac2d397f5404ea9aea1c997fb2cdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:07.436151 2110 catalog_manager.cc:4841] TS fae69f4e3351498ba7cc458f3e8e5174 (127.0.98.3:36419): tablet 58ab2b7f53464929ac76b61a6d5b3f1c (table test-table [id=1d1676e65ee94e7494d38769e5270de9]) successfully deleted
W20260925 16:27:07.436992 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33793 (error 108)
I20260925 16:27:07.437201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 2335
I20260925 16:27:07.440117 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:07.439494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33793 (local address 127.0.0.1:56230)
0925 16:27:07.439885 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:07.439969 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
W20260925 16:27:07.440289 2110 catalog_manager.cc:4584] TS cd9f0f0ecfeb458da9c672a3a428a7b6 (127.0.98.1:33793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58ab2b7f53464929ac76b61a6d5b3f1c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33793: connect: Connection refused (error 111)
I20260925 16:27:07.445638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 2475
I20260925 16:27:07.445729 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:07.445411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33067 (local address 127.0.0.1:57574)
0925 16:27:07.445562 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:07.445618 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 16:27:07.454087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 2091
2026-09-25T16:27:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6470 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:07 dist-test-slave-31kt krb5kdc[2780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:07 dist-test-slave-31kt krb5kdc[2780](info): set up 2 sockets
Sep 25 16:27:07 dist-test-slave-31kt krb5kdc[2780](info): commencing operation
krb5kdc: starting...
W20260925 16:27:08.430079 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:08 dist-test-slave-31kt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:08.573333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:08.712277 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:08.712885 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:08.712984 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:08.719233 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:08.719520 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:08.719600 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:08.719640 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:08.719676 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:08.727723 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:08.729465 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:08.731247 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:08.744357 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:08.744448 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:08.744578 2793 server_base.cc:1034] running on GCE node
W20260925 16:27:08.744448 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:08.746176 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:08.747925 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:08.749161 2793 hybrid_clock.cc:648] HybridClock initialized: now 1790353628749119 us; error 78 us; skew 500 ppm
Sep 25 16:27:08 dist-test-slave-31kt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:08.758100 2793 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:08.759677 2793 webserver.cc:460] Webserver started at http://127.0.98.62:41093/ using document root <none> and password file <none>
I20260925 16:27:08.760484 2793 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:08.760566 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:08.760823 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:08.764269 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "76c37aa5e14a467abe1142cedeeca569"
format_stamp: "Formatted at 2026-09-25 16:27:08 on dist-test-slave-31kt"
server_key: "588c6dc34acafeb2c760d765d7562872"
server_key_iv: "ef9c87de6c801335b64721424b46cc09"
server_key_version: "encryptionkey@0"
I20260925 16:27:08.765230 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "76c37aa5e14a467abe1142cedeeca569"
format_stamp: "Formatted at 2026-09-25 16:27:08 on dist-test-slave-31kt"
server_key: "588c6dc34acafeb2c760d765d7562872"
server_key_iv: "ef9c87de6c801335b64721424b46cc09"
server_key_version: "encryptionkey@0"
I20260925 16:27:08.771361 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 16:27:08.775933 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:08.778143 2793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:27:08.778373 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "76c37aa5e14a467abe1142cedeeca569"
format_stamp: "Formatted at 2026-09-25 16:27:08 on dist-test-slave-31kt"
server_key: "588c6dc34acafeb2c760d765d7562872"
server_key_iv: "ef9c87de6c801335b64721424b46cc09"
server_key_version: "encryptionkey@0"
I20260925 16:27:08.778589 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:08.806908 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:08.812546 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:08.812856 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:08.824674 2793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41237
I20260925 16:27:08.824947 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41237 every 8 connection(s)
I20260925 16:27:08.826318 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:08.832352 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:08.833562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 2793
I20260925 16:27:08.833746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:08.834126 392 external_mini_cluster.cc:1428] Setting key 72a647e960e0d498ed4afd4ffd7c0258
I20260925 16:27:08.844170 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569: Bootstrap starting.
Sep 25 16:27:08 dist-test-slave-31kt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:08.848460 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:08.849746 2861 log.cc:824] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:08.854452 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569: No bootstrap required, opened a new log
I20260925 16:27:08.857993 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:08.836326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.0.98.62:41237)
0925 16:27:08.836997 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:08.837020 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:08.837076 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:27:08.838063 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:08.838093 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:08.838672 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:08.839159 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:08.840413 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:08.841897 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:08.843725 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:08.844606 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:08.844816 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:08.849037 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:08.849059 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:08.849089 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:08.849153 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:08.852331 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:08.853628 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:08.853639 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:08.853649 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:08.853966 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:08.854579 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:08.854583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:08.854585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:08.855344 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:08.855614 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:08.856030 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 16:27:08.856475 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":205,"threads_started":1}
I20260925 16:27:08.860122 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c37aa5e14a467abe1142cedeeca569" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41237 } }
I20260925 16:27:08.860476 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:08.860553 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c37aa5e14a467abe1142cedeeca569, State: Initialized, Role: FOLLOWER
I20260925 16:27:08.861112 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c37aa5e14a467abe1142cedeeca569" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41237 } }
I20260925 16:27:08.861280 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:08.861356 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:08.861467 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:08.862859 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c37aa5e14a467abe1142cedeeca569" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41237 } }
I20260925 16:27:08.863327 2861 leader_election.cc:302] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c37aa5e14a467abe1142cedeeca569; no voters:
I20260925 16:27:08.863754 2861 leader_election.cc:288] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:08.864055 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:08.864408 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [term 1 LEADER]: Becoming Leader. State: Replica: 76c37aa5e14a467abe1142cedeeca569, State: Running, Role: LEADER
I20260925 16:27:08.864905 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c37aa5e14a467abe1142cedeeca569" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41237 } }
I20260925 16:27:08.865320 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:08.867148 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c37aa5e14a467abe1142cedeeca569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c37aa5e14a467abe1142cedeeca569" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41237 } } }
I20260925 16:27:08.867202 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c37aa5e14a467abe1142cedeeca569. Latest consensus state: current_term: 1 leader_uuid: "76c37aa5e14a467abe1142cedeeca569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c37aa5e14a467abe1142cedeeca569" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41237 } } }
I20260925 16:27:08.867363 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:08.867419 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569 [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:08.872678 2881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:08.872875 2881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:08.873050 2878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:08.874341 2878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:08.883153 2878 catalog_manager.cc:1347] Generated new cluster ID: 30390217c3c744e5beff5b4011ac1dda
I20260925 16:27:08.883270 2878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:08.896123 2878 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:08.897315 2878 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:08.905026 2878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76c37aa5e14a467abe1142cedeeca569: Generated new TSK 0
I20260925 16:27:08.906432 2878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:27:08.964084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41237
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:09.123106 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:09.123432 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:09.123479 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:09.128212 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:09.128377 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:09.128463 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:09.137746 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:09.139402 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:09.140707 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:09.152901 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:09.152935 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:09.152935 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:09.153376 2887 server_base.cc:1034] running on GCE node
I20260925 16:27:09.154079 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:09.154839 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:09.156083 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790353629156052 us; error 55 us; skew 500 ppm
Sep 25 16:27:09 dist-test-slave-31kt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:09.159762 2887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:09.161628 2887 webserver.cc:460] Webserver started at http://127.0.98.1:44785/ using document root <none> and password file <none>
I20260925 16:27:09.162477 2887 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:09.162562 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:09.162801 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:09.166237 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "75f7300184f04041a82f0c66fc68001f"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "548346d7f3e3e9e155ad4695670808f0"
server_key_iv: "640578dc05e51250a65be58eb05048b3"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.167102 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75f7300184f04041a82f0c66fc68001f"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "548346d7f3e3e9e155ad4695670808f0"
server_key_iv: "640578dc05e51250a65be58eb05048b3"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.172122 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:09.175663 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.177881 2887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:09.178057 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "75f7300184f04041a82f0c66fc68001f"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "548346d7f3e3e9e155ad4695670808f0"
server_key_iv: "640578dc05e51250a65be58eb05048b3"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.178238 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:09.198521 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:09.203090 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:09.203343 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:09.204154 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:09.205323 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:09.205381 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.205432 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:09.205468 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.217034 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42681
I20260925 16:27:09.217113 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42681 every 8 connection(s)
I20260925 16:27:09.218729 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:09.225504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 2887
I20260925 16:27:09.225656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:09.226042 392 external_mini_cluster.cc:1428] Setting key 7ea96cfdd9c9c3cb7f876cbf4d2222da
Sep 25 16:27:09 dist-test-slave-31kt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:27:09.240100 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:09.221511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36441 (local address 127.0.98.62:41237)
0925 16:27:09.221671 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:09.221676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:09.223234 (+ 1558us) server_negotiation.cc:407] Connection header received
0925 16:27:09.224992 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:09.224996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:09.225062 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:09.225163 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:09.227298 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.228037 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.229553 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.229845 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.229965 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:09.234111 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:09.234116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:09.234118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:09.234158 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:09.236442 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:09.237584 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:09.237588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:09.237590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:09.237728 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:09.238427 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:09.238432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:09.238433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:09.238746 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:09.238848 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:09.239750 (+ 902us) server_negotiation.cc:299] Negotiation successful
0925 16:27:09.239943 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260925 16:27:09.241122 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:09.222206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41237 (local address 127.0.98.1:36441)
0925 16:27:09.222923 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:09.223031 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:09.224643 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:09.225509 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:09.225522 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:09.226217 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:09.227032 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:09.227055 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.228273 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.228278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:09.229369 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:09.229387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.230046 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.230052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:09.230263 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:09.231189 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:09.231209 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:09.233660 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:09.236709 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:09.236734 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:09.236767 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:09.237367 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:09.237906 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:09.237911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:09.237913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:09.238201 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:09.238875 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:09.238885 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:09.239273 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 16:27:09.239499 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 16:27:09.239934 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":195,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:27:09.242733 3016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41237
I20260925 16:27:09.243062 3016 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:09.243796 3016 heartbeater.cc:511] Master 127.0.98.62:41237 requested a full tablet report, sending...
I20260925 16:27:09.245903 2816 ts_manager.cc:194] Registered new tserver with Master: 75f7300184f04041a82f0c66fc68001f (127.0.98.1:42681)
I20260925 16:27:09.247231 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:09.263429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41237
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:09.405992 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:09.406422 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:09.406493 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:09.411432 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:09.411537 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:09.411636 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:09.418478 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:09.420084 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:09.421576 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:09.434386 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:09.434459 3021 server_base.cc:1034] running on GCE node
W20260925 16:27:09.434423 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:09.434758 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:09.435395 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:09.436266 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:09.437479 3021 hybrid_clock.cc:648] HybridClock initialized: now 1790353629437449 us; error 54 us; skew 500 ppm
Sep 25 16:27:09 dist-test-slave-31kt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:09.441028 3021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:09.442364 3021 webserver.cc:460] Webserver started at http://127.0.98.2:35907/ using document root <none> and password file <none>
I20260925 16:27:09.443091 3021 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:09.443164 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:09.443410 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:09.446255 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "190040fb9be444788c4edf3778f94ea9"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "609aa16dea1170406424a5115bf22908"
server_key_iv: "f32c486bd7b3cd72e3af0a8bddc57d30"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.447013 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "190040fb9be444788c4edf3778f94ea9"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "609aa16dea1170406424a5115bf22908"
server_key_iv: "f32c486bd7b3cd72e3af0a8bddc57d30"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.452454 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 16:27:09.456225 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.458259 3021 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:09.458487 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "190040fb9be444788c4edf3778f94ea9"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "609aa16dea1170406424a5115bf22908"
server_key_iv: "f32c486bd7b3cd72e3af0a8bddc57d30"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.458648 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:09.478531 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:09.482822 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:09.483074 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:09.483911 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:09.485174 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:09.485237 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.485306 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:09.485332 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.498377 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39603
I20260925 16:27:09.498411 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39603 every 8 connection(s)
I20260925 16:27:09.499635 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:09.502847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3021
I20260925 16:27:09.502983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:09.503252 392 external_mini_cluster.cc:1428] Setting key 4ab08b47c03b5a6a4e0e8f3b71d80322
Sep 25 16:27:09 dist-test-slave-31kt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:09.519728 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:09.502412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44283 (local address 127.0.98.62:41237)
0925 16:27:09.502675 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:09.502680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:09.503937 (+ 1257us) server_negotiation.cc:407] Connection header received
0925 16:27:09.505821 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:09.505825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:09.505907 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:09.505986 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:09.507874 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.508682 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.510710 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.511007 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.511120 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:09.514743 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:09.514747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:09.514749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:09.514785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:09.516541 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:09.517310 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:09.517313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:09.517315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:09.517404 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:09.517896 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:09.517903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:09.517904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:09.518252 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:09.518391 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:09.519397 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0925 16:27:09.519576 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 16:27:09.520996 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:09.502897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41237 (local address 127.0.98.2:44283)
0925 16:27:09.503703 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:09.503794 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:09.505431 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:09.506296 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:09.506309 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:09.506891 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:09.507670 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:09.507689 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.508871 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.508875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:09.510493 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:09.510515 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.511246 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.511251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:09.511513 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:09.512323 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:09.512348 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:09.514469 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:09.516727 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:09.516737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:09.516761 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:09.517154 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:09.517501 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:09.517504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:09.517506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:09.517689 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:09.518422 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:09.518433 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:09.518972 (+ 539us) client_negotiation.cc:769] Sending connection context
0925 16:27:09.519269 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 16:27:09.519795 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":173,"threads_started":1}
I20260925 16:27:09.522562 3150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41237
I20260925 16:27:09.523005 3150 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:09.523833 3150 heartbeater.cc:511] Master 127.0.98.62:41237 requested a full tablet report, sending...
I20260925 16:27:09.525503 2816 ts_manager.cc:194] Registered new tserver with Master: 190040fb9be444788c4edf3778f94ea9 (127.0.98.2:39603)
I20260925 16:27:09.526378 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:09.539057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41237
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:09.673257 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:09.673604 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:09.673655 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:09.678556 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:09.678648 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:09.678711 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:09.686868 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:09.688213 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:09.689721 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:09.702169 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:09.702168 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:09.703282 3155 server_base.cc:1034] running on GCE node
W20260925 16:27:09.702215 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:09.704643 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:09.705895 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:09.707118 3155 hybrid_clock.cc:648] HybridClock initialized: now 1790353629707095 us; error 69 us; skew 500 ppm
Sep 25 16:27:09 dist-test-slave-31kt krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:09.711735 3155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:09.713156 3155 webserver.cc:460] Webserver started at http://127.0.98.3:33785/ using document root <none> and password file <none>
I20260925 16:27:09.714048 3155 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:09.714156 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:09.714442 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:09.717494 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2e81a3258f34ce380d82bbb94b0f2a8"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "83d015e765bb987c3390d36c5f52aa66"
server_key_iv: "551320c3e7a9a92490a4a584057c5fb6"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.718439 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2e81a3258f34ce380d82bbb94b0f2a8"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "83d015e765bb987c3390d36c5f52aa66"
server_key_iv: "551320c3e7a9a92490a4a584057c5fb6"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.723793 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:09.727643 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.729655 3155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:09.729930 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "f2e81a3258f34ce380d82bbb94b0f2a8"
format_stamp: "Formatted at 2026-09-25 16:27:09 on dist-test-slave-31kt"
server_key: "83d015e765bb987c3390d36c5f52aa66"
server_key_iv: "551320c3e7a9a92490a4a584057c5fb6"
server_key_version: "encryptionkey@0"
I20260925 16:27:09.730098 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:09.747722 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:09.752573 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:09.752818 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:09.753626 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:09.754786 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:09.754842 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.754906 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:09.754938 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:09.767484 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46355
I20260925 16:27:09.767542 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46355 every 8 connection(s)
I20260925 16:27:09.768800 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:09.778518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3155
I20260925 16:27:09.778687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:09.779023 392 external_mini_cluster.cc:1428] Setting key a9fa3fcd4f91b25619baf9467578804c
Sep 25 16:27:09 dist-test-slave-31kt krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:09.788949 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:09.771423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56569 (local address 127.0.98.62:41237)
0925 16:27:09.771614 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:09.771618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:09.772904 (+ 1286us) server_negotiation.cc:407] Connection header received
0925 16:27:09.775091 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:09.775095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:09.775230 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:09.775354 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:09.777407 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.778156 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.780020 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.780361 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.780489 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:09.783767 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:09.783772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:09.783775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:09.783818 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:09.786041 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:09.786853 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:09.786857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:09.786859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:09.786993 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:09.787496 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:09.787499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:09.787500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:09.787749 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:09.787843 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:09.788568 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 16:27:09.788772 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:27:09.790053 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:09.771901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41237 (local address 127.0.98.3:56569)
0925 16:27:09.772655 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:09.772723 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:09.774704 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:09.775722 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:09.775735 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:09.776350 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:09.777137 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:09.777164 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.778446 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.778452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:09.779826 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:09.779850 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:09.780623 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:09.780629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:09.780882 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:09.781596 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:09.781617 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:09.783448 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:09.786256 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:09.786266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:09.786292 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:09.786694 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:09.787161 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:09.787164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:09.787166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:09.787371 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:09.787878 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:09.787885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:09.788247 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 16:27:09.788474 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 16:27:09.788838 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":153,"threads_started":1}
I20260925 16:27:09.791942 3284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41237
I20260925 16:27:09.792310 3284 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:09.793138 3284 heartbeater.cc:511] Master 127.0.98.62:41237 requested a full tablet report, sending...
I20260925 16:27:09.794754 2816 ts_manager.cc:194] Registered new tserver with Master: f2e81a3258f34ce380d82bbb94b0f2a8 (127.0.98.3:46355)
I20260925 16:27:09.795498 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56569
I20260925 16:27:09.807770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:27:09.824105 2864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:09.822100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.0.98.62:41237)
0925 16:27:09.822309 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:09.822315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:09.822538 (+ 223us) server_negotiation.cc:407] Connection header received
0925 16:27:09.822716 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:09.822719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:09.822772 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:09.822851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:09.823943 (+ 1092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35856: BlockingRecv error: recv got EOF from 127.0.0.1:35856 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:27:09.825644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 2887
I20260925 16:27:09.832011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3021
I20260925 16:27:09.840195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3155
I20260925 16:27:09.847105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 2793
2026-09-25T16:27:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2391 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:10 dist-test-slave-31kt krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:10 dist-test-slave-31kt krb5kdc[3296](info): set up 2 sockets
Sep 25 16:27:10 dist-test-slave-31kt krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260925 16:27:10.926151 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:11 dist-test-slave-31kt krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:11.070156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:11.200646 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:11.201141 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:11.201213 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:11.207261 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:11.207378 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:11.207412 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:11.207436 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:11.207458 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:11.215008 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:11.216723 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:11.218294 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:11.230013 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:11.230060 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:11.230177 3309 server_base.cc:1034] running on GCE node
W20260925 16:27:11.230060 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:11.231410 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:11.233397 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:11.234678 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790353631234624 us; error 110 us; skew 500 ppm
Sep 25 16:27:11 dist-test-slave-31kt krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:11.238722 3309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:11.240077 3309 webserver.cc:460] Webserver started at http://127.0.98.62:34079/ using document root <none> and password file <none>
I20260925 16:27:11.240855 3309 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:11.240947 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:11.241220 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:11.244410 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "a7f6d7e6e3804185bc67be93868a8f71"
format_stamp: "Formatted at 2026-09-25 16:27:11 on dist-test-slave-31kt"
server_key: "e501fa8d87f9d30d87ae59fc61ecd453"
server_key_iv: "57013832aac62aaa57ed860e834ade91"
server_key_version: "encryptionkey@0"
I20260925 16:27:11.245318 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7f6d7e6e3804185bc67be93868a8f71"
format_stamp: "Formatted at 2026-09-25 16:27:11 on dist-test-slave-31kt"
server_key: "e501fa8d87f9d30d87ae59fc61ecd453"
server_key_iv: "57013832aac62aaa57ed860e834ade91"
server_key_version: "encryptionkey@0"
I20260925 16:27:11.252084 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 16:27:11.256558 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:11.258870 3309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:27:11.259137 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "a7f6d7e6e3804185bc67be93868a8f71"
format_stamp: "Formatted at 2026-09-25 16:27:11 on dist-test-slave-31kt"
server_key: "e501fa8d87f9d30d87ae59fc61ecd453"
server_key_iv: "57013832aac62aaa57ed860e834ade91"
server_key_version: "encryptionkey@0"
I20260925 16:27:11.259322 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:11.272928 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:11.276970 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:11.277192 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:11.285306 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34259
I20260925 16:27:11.285615 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34259 every 8 connection(s)
I20260925 16:27:11.286571 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:11.290162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3309
I20260925 16:27:11.290282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:11.290565 392 external_mini_cluster.cc:1428] Setting key cf2bd0a7add3f927ad8473d64bc6fe79
I20260925 16:27:11.291103 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:11.301429 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71: Bootstrap starting.
Sep 25 16:27:11 dist-test-slave-31kt krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:11.305469 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:11.306603 3377 log.cc:824] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:11.309348 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71: No bootstrap required, opened a new log
I20260925 16:27:11.310958 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:11.292393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.0.98.62:34259)
0925 16:27:11.293195 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:11.293215 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:11.293504 (+ 289us) server_negotiation.cc:407] Connection header received
0925 16:27:11.294503 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:11.294531 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:11.294995 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:11.295365 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:11.296677 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:11.298375 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:11.299958 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:11.300653 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:11.300860 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:11.304841 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:11.304850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:11.304880 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:11.304933 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:11.307385 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:11.308021 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:11.308028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:11.308035 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:11.308195 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:11.308480 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:11.308483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:11.308485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:11.309031 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:11.309237 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:11.309596 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 16:27:11.310015 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":254,"threads_started":1}
I20260925 16:27:11.314469 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f6d7e6e3804185bc67be93868a8f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34259 } }
I20260925 16:27:11.314872 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:11.314949 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f6d7e6e3804185bc67be93868a8f71, State: Initialized, Role: FOLLOWER
I20260925 16:27:11.315538 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:27:11.315546 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f6d7e6e3804185bc67be93868a8f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34259 } }
I20260925 16:27:11.315706 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:11.315788 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:11.315913 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:11.317171 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f6d7e6e3804185bc67be93868a8f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34259 } }
I20260925 16:27:11.317667 3377 leader_election.cc:302] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f6d7e6e3804185bc67be93868a8f71; no voters:
I20260925 16:27:11.318218 3377 leader_election.cc:288] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:11.318473 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:11.318814 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [term 1 LEADER]: Becoming Leader. State: Replica: a7f6d7e6e3804185bc67be93868a8f71, State: Running, Role: LEADER
I20260925 16:27:11.319294 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f6d7e6e3804185bc67be93868a8f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34259 } }
I20260925 16:27:11.320031 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:11.321430 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f6d7e6e3804185bc67be93868a8f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f6d7e6e3804185bc67be93868a8f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34259 } } }
I20260925 16:27:11.321620 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:11.322062 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f6d7e6e3804185bc67be93868a8f71. Latest consensus state: current_term: 1 leader_uuid: "a7f6d7e6e3804185bc67be93868a8f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f6d7e6e3804185bc67be93868a8f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34259 } } }
I20260925 16:27:11.322180 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:11.322644 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:11.327415 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:11.336637 3392 catalog_manager.cc:1347] Generated new cluster ID: 70a3177793284819bfe1ae1e3dd9a4bf
I20260925 16:27:11.336746 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:11.338443 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:11.331974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.0.98.62:34259)
0925 16:27:11.332380 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:11.332386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:11.332493 (+ 107us) server_negotiation.cc:407] Connection header received
0925 16:27:11.332663 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:11.332667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:11.332736 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:11.332870 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:11.333666 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:11.334451 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:11.335425 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:11.335708 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:11.335818 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:11.337056 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:11.337062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:11.337066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:11.337803 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:11.337864 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:11.337891 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:11.338031 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:11.338179 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 16:27:11.338273 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260925 16:27:11.350162 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:11.351565 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:11.370363 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f6d7e6e3804185bc67be93868a8f71: Generated new TSK 0
I20260925 16:27:11.371506 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:11.380158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3309
2026-09-25T16:27:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1531 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:11 dist-test-slave-31kt krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:11 dist-test-slave-31kt krb5kdc[3410](info): set up 2 sockets
Sep 25 16:27:11 dist-test-slave-31kt krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260925 16:27:12.354264 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:12 dist-test-slave-31kt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:12.497473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:12.634011 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:12.634413 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:12.634477 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:12.639057 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:12.639160 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:12.639189 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:12.639212 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:12.639235 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:12.645566 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:12.647119 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:12.648442 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:27:12.659842 3423 server_base.cc:1034] running on GCE node
W20260925 16:27:12.659786 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:12.659724 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:12.659724 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:12.660868 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:12.662617 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:12.663820 3423 hybrid_clock.cc:648] HybridClock initialized: now 1790353632663792 us; error 67 us; skew 500 ppm
Sep 25 16:27:12 dist-test-slave-31kt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:12.667411 3423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:12.668653 3423 webserver.cc:460] Webserver started at http://127.0.98.62:33667/ using document root <none> and password file <none>
I20260925 16:27:12.669368 3423 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:12.669441 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:12.669687 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:12.672528 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "cde407b5026d45a09bcef6281655a080"
format_stamp: "Formatted at 2026-09-25 16:27:12 on dist-test-slave-31kt"
server_key: "9657c400c117809e932e26d13e5c508d"
server_key_iv: "ea25576841fa9322b821d4db1d6ebb29"
server_key_version: "encryptionkey@0"
I20260925 16:27:12.673482 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "cde407b5026d45a09bcef6281655a080"
format_stamp: "Formatted at 2026-09-25 16:27:12 on dist-test-slave-31kt"
server_key: "9657c400c117809e932e26d13e5c508d"
server_key_iv: "ea25576841fa9322b821d4db1d6ebb29"
server_key_version: "encryptionkey@0"
I20260925 16:27:12.680788 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 16:27:12.684906 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:12.686940 3423 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:12.687201 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "cde407b5026d45a09bcef6281655a080"
format_stamp: "Formatted at 2026-09-25 16:27:12 on dist-test-slave-31kt"
server_key: "9657c400c117809e932e26d13e5c508d"
server_key_iv: "ea25576841fa9322b821d4db1d6ebb29"
server_key_version: "encryptionkey@0"
I20260925 16:27:12.687362 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:12.711297 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:12.715466 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:12.715713 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:12.725035 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41669
I20260925 16:27:12.725343 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41669 every 8 connection(s)
I20260925 16:27:12.726361 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:12.730178 3491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:12.736479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3423
I20260925 16:27:12.736639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:12.736837 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080: Bootstrap starting.
I20260925 16:27:12.736943 392 external_mini_cluster.cc:1428] Setting key bc7dee2aeb3daab4b9040cfb14767aa7
I20260925 16:27:12.739964 3491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:12.741318 3491 log.cc:824] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:12.744596 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080: No bootstrap required, opened a new log
I20260925 16:27:12.748932 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde407b5026d45a09bcef6281655a080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260925 16:27:12.749208 3491 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:12.749243 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde407b5026d45a09bcef6281655a080, State: Initialized, Role: FOLLOWER
Sep 25 16:27:12 dist-test-slave-31kt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:12.749816 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde407b5026d45a09bcef6281655a080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260925 16:27:12.750018 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:12.750100 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:12.750211 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:12.751241 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde407b5026d45a09bcef6281655a080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260925 16:27:12.751645 3491 leader_election.cc:302] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde407b5026d45a09bcef6281655a080; no voters:
I20260925 16:27:12.752014 3491 leader_election.cc:288] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:12.752215 3496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:12.752542 3496 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [term 1 LEADER]: Becoming Leader. State: Replica: cde407b5026d45a09bcef6281655a080, State: Running, Role: LEADER
I20260925 16:27:12.753026 3496 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde407b5026d45a09bcef6281655a080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260925 16:27:12.753850 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:12.755460 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde407b5026d45a09bcef6281655a080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde407b5026d45a09bcef6281655a080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } } }
I20260925 16:27:12.755637 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:12.755976 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde407b5026d45a09bcef6281655a080. Latest consensus state: current_term: 1 leader_uuid: "cde407b5026d45a09bcef6281655a080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde407b5026d45a09bcef6281655a080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } } }
I20260925 16:27:12.756080 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:12.756510 3505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:12.758159 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:12.738884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.0.98.62:41669)
0925 16:27:12.739518 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:12.739558 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:12.739612 (+ 54us) server_negotiation.cc:407] Connection header received
0925 16:27:12.740844 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:12.740880 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:12.741554 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:12.742262 (+ 708us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:12.743431 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:12.744986 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:12.746323 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:12.747091 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:12.747291 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:12.750438 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:12.750447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:12.750472 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:12.750522 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:12.752792 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:12.753506 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:12.753514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:12.753521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:12.753705 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:12.754076 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:12.754080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:12.754081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:12.754823 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:12.755960 (+ 1137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:12.756302 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:27:12.756806 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":424,"thread_start_us":181,"threads_started":1}
I20260925 16:27:12.761359 3505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:12.769418 3505 catalog_manager.cc:1347] Generated new cluster ID: 8a8454f67e4a4bb8b0e4ea1ad9c3f43e
I20260925 16:27:12.769542 3505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:12.787626 3505 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:12.789165 3505 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:12.800704 3505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde407b5026d45a09bcef6281655a080: Generated new TSK 0
I20260925 16:27:12.801959 3505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:27:12.837733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:12.974484 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:12.974871 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:12.974927 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:12.979965 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:12.980067 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:12.980131 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:12.987003 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:12.988567 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:12.990188 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:13.005787 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:13.005806 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:13.005825 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:13.005926 3517 server_base.cc:1034] running on GCE node
I20260925 16:27:13.006831 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:13.007804 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:13.009014 3517 hybrid_clock.cc:648] HybridClock initialized: now 1790353633008986 us; error 55 us; skew 500 ppm
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:13.013094 3517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:13.014400 3517 webserver.cc:460] Webserver started at http://127.0.98.1:44997/ using document root <none> and password file <none>
I20260925 16:27:13.015321 3517 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:13.015502 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:13.015769 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:13.018685 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "7880c4eeaf2043caa0f1088439e32e2a"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "c411d12ec7c74d155c344ca84a889254"
server_key_iv: "6149ddde2ef7e7983f49876b3d220d36"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.019352 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7880c4eeaf2043caa0f1088439e32e2a"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "c411d12ec7c74d155c344ca84a889254"
server_key_iv: "6149ddde2ef7e7983f49876b3d220d36"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.024662 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:13.028460 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.030740 3517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:13.030962 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "7880c4eeaf2043caa0f1088439e32e2a"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "c411d12ec7c74d155c344ca84a889254"
server_key_iv: "6149ddde2ef7e7983f49876b3d220d36"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.031157 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:13.080569 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:13.085520 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:13.085974 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:13.086846 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:13.087970 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:13.088028 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.088090 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:13.088121 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.101228 3517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36119
I20260925 16:27:13.101313 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36119 every 8 connection(s)
I20260925 16:27:13.102551 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:13.106886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3517
I20260925 16:27:13.107039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:13.107364 392 external_mini_cluster.cc:1428] Setting key ee3bfb04eded673f761e668260a2b87e
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:27:13.123950 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.105383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49399 (local address 127.0.98.62:41669)
0925 16:27:13.105744 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:13.105750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:13.107175 (+ 1425us) server_negotiation.cc:407] Connection header received
0925 16:27:13.109122 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:13.109126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:13.109199 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:13.109281 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:13.111577 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.112209 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.114390 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.114665 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.114786 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.118766 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:13.118770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:13.118772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:13.118814 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:13.120610 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.121500 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.121506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.121508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.121674 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.122189 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.122192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.122193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.122469 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:13.122584 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:13.123505 (+ 921us) server_negotiation.cc:299] Negotiation successful
0925 16:27:13.123733 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260925 16:27:13.125242 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.106153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41669 (local address 127.0.98.1:49399)
0925 16:27:13.106898 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:13.106993 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:13.108792 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:13.109568 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:13.109578 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:13.110124 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:13.111259 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:13.111288 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.112418 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.112422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:13.114162 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:13.114183 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.114914 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.114919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:13.115217 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.116094 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:13.116124 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:13.118418 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:13.120825 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:13.120833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:13.120857 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:13.121299 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:13.121825 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:13.121829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:13.121831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:13.122058 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:13.122625 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:13.122634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:13.123045 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 16:27:13.123332 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 16:27:13.123830 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":189,"threads_started":1}
I20260925 16:27:13.127463 3646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41669
I20260925 16:27:13.127928 3646 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:13.128949 3646 heartbeater.cc:511] Master 127.0.98.62:41669 requested a full tablet report, sending...
I20260925 16:27:13.131558 3455 ts_manager.cc:194] Registered new tserver with Master: 7880c4eeaf2043caa0f1088439e32e2a (127.0.98.1:36119)
I20260925 16:27:13.132921 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:13.142645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:13.281553 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:13.281947 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:13.282003 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:13.287712 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:13.287838 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:13.287930 3651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:13.296192 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:13.297528 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:13.298707 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:13.312565 3657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:13.312492 3656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:13.312841 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:13.313162 3651 server_base.cc:1034] running on GCE node
I20260925 16:27:13.313947 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:13.314898 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:13.316093 3651 hybrid_clock.cc:648] HybridClock initialized: now 1790353633316064 us; error 60 us; skew 500 ppm
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:13.320137 3651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:13.321645 3651 webserver.cc:460] Webserver started at http://127.0.98.2:33579/ using document root <none> and password file <none>
I20260925 16:27:13.322523 3651 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:13.322625 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:13.322917 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:13.326102 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "3004c4d5f59b42bdb6ae1fb023197450"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "e88fa5eba71b30c632cdf2fe7ba08270"
server_key_iv: "cf9da78265568a0124787c3e63c3482f"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.327049 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3004c4d5f59b42bdb6ae1fb023197450"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "e88fa5eba71b30c632cdf2fe7ba08270"
server_key_iv: "cf9da78265568a0124787c3e63c3482f"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.333299 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 16:27:13.337648 3666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.339715 3651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 16:27:13.339923 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "3004c4d5f59b42bdb6ae1fb023197450"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "e88fa5eba71b30c632cdf2fe7ba08270"
server_key_iv: "cf9da78265568a0124787c3e63c3482f"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.340116 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:13.365630 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:13.370818 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:13.371095 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:13.372009 3651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:13.373215 3651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:13.373289 3651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.373339 3651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:13.373427 3651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.386317 3651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42865
I20260925 16:27:13.386407 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42865 every 8 connection(s)
I20260925 16:27:13.387826 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:13.392614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3651
I20260925 16:27:13.392759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:13.393097 392 external_mini_cluster.cc:1428] Setting key c2a58fc18d311aec18e7d8d4518aa85a
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:13.409164 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.390786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36173 (local address 127.0.98.62:41669)
0925 16:27:13.390993 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:13.390998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:13.392346 (+ 1348us) server_negotiation.cc:407] Connection header received
0925 16:27:13.394295 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:13.394299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:13.394363 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:13.394436 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:13.396454 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.397161 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.399064 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.399460 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.399613 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.403660 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:13.403665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:13.403669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:13.403716 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:13.405533 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.406431 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.406435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.406436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.406551 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.407228 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.407234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.407236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.407527 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:13.407627 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:13.408749 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0925 16:27:13.408914 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 16:27:13.410508 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.391376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41669 (local address 127.0.98.2:36173)
0925 16:27:13.392133 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:13.392195 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:13.394010 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:13.394839 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:13.394860 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:13.395467 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:13.396154 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:13.396177 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.397413 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.397418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:13.398759 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:13.398785 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.399726 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.399731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:13.399955 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.400914 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:13.400938 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:13.403326 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:13.405714 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:13.405723 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:13.405746 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:13.406212 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:13.406747 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:13.406751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:13.406753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:13.407029 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:13.407682 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:13.407697 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:13.408249 (+ 552us) client_negotiation.cc:769] Sending connection context
0925 16:27:13.408553 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 16:27:13.409146 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":233,"threads_started":1}
I20260925 16:27:13.412274 3780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41669
I20260925 16:27:13.412626 3780 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:13.413362 3780 heartbeater.cc:511] Master 127.0.98.62:41669 requested a full tablet report, sending...
I20260925 16:27:13.415083 3455 ts_manager.cc:194] Registered new tserver with Master: 3004c4d5f59b42bdb6ae1fb023197450 (127.0.98.2:42865)
I20260925 16:27:13.415865 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:13.426559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:13.567828 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:13.568284 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:13.568351 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:13.573268 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:13.573395 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:13.573490 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:13.580325 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:13.582001 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:13.583529 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:13.598089 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:13.598093 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:13.598093 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:13.598206 3785 server_base.cc:1034] running on GCE node
I20260925 16:27:13.598991 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:13.599884 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:13.601147 3785 hybrid_clock.cc:648] HybridClock initialized: now 1790353633601110 us; error 60 us; skew 500 ppm
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:13.605482 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:13.606804 3785 webserver.cc:460] Webserver started at http://127.0.98.3:37255/ using document root <none> and password file <none>
I20260925 16:27:13.607685 3785 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:13.607788 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:13.608031 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:13.610939 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "14192e124b6c462c853a3e626775eaaa"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "196b99ca2e3011ccb4a8c5d260ebe4fb"
server_key_iv: "a9b477c94be7f3769ef884206527b405"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.611657 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14192e124b6c462c853a3e626775eaaa"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "196b99ca2e3011ccb4a8c5d260ebe4fb"
server_key_iv: "a9b477c94be7f3769ef884206527b405"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.616873 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:13.620978 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.622951 3785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:13.623133 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "14192e124b6c462c853a3e626775eaaa"
format_stamp: "Formatted at 2026-09-25 16:27:13 on dist-test-slave-31kt"
server_key: "196b99ca2e3011ccb4a8c5d260ebe4fb"
server_key_iv: "a9b477c94be7f3769ef884206527b405"
server_key_version: "encryptionkey@0"
I20260925 16:27:13.623273 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:13.638136 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:13.642831 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:13.643088 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:13.643889 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:13.645079 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:13.645143 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.645195 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:13.645208 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:13.656683 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36595
I20260925 16:27:13.656731 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36595 every 8 connection(s)
I20260925 16:27:13.657961 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:13.666538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3785
I20260925 16:27:13.666705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:13.667028 392 external_mini_cluster.cc:1428] Setting key 3341b3e0041a3be69e82eff84ac1ced1
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:13.678081 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.660913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51413 (local address 127.0.98.62:41669)
0925 16:27:13.661150 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:13.661155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:13.662614 (+ 1459us) server_negotiation.cc:407] Connection header received
0925 16:27:13.664460 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:13.664464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:13.664699 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:13.664859 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:13.666610 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.667238 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.668717 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.669097 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.669240 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.672900 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:13.672905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:13.672908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:13.672948 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:13.675131 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.675975 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.675978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.675980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.676086 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.676524 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.676528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.676529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.676787 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:13.676892 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:13.677633 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 16:27:13.677870 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 16:27:13.679339 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.661504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41669 (local address 127.0.98.3:51413)
0925 16:27:13.662398 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:13.662455 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:13.664161 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:13.665227 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:13.665239 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:13.665732 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:13.666394 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:13.666412 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.667381 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.667385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:13.668555 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:13.668571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.669364 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.669369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:13.669715 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.670550 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:13.670570 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:13.672470 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:13.675356 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:13.675365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:13.675392 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:13.675830 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:13.676193 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:13.676195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:13.676197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:13.676413 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:13.676906 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:13.676917 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:13.677310 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 16:27:13.677535 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 16:27:13.678073 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":324,"threads_started":1}
I20260925 16:27:13.681720 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41669
I20260925 16:27:13.682099 3914 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:13.682935 3914 heartbeater.cc:511] Master 127.0.98.62:41669 requested a full tablet report, sending...
I20260925 16:27:13.684532 3454 ts_manager.cc:194] Registered new tserver with Master: 14192e124b6c462c853a3e626775eaaa (127.0.98.3:36595)
I20260925 16:27:13.685232 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51413
I20260925 16:27:13.697680 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353633, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:13.734005 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:13.722057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.0.98.62:41669)
0925 16:27:13.722242 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:13.722248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:13.722272 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:27:13.722477 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:13.722483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:13.722555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:13.722752 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:13.723836 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.724578 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.725678 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:13.726004 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:13.726117 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:13.729980 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:13.729984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:13.729987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:13.730037 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:13.731901 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.732535 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.732539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.732540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.732639 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:13.732911 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:13.732915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:13.732917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:13.733223 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:13.733369 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:13.733618 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 16:27:13.733807 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 16:27:13.734496 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49442
I20260925 16:27:13.735667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3517
I20260925 16:27:13.743500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3651
I20260925 16:27:13.749737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3785
I20260925 16:27:13.756698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3423
2026-09-25T16:27:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2377 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3926](info): set up 2 sockets
Sep 25 16:27:13 dist-test-slave-31kt krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260925 16:27:14.746107 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:14 dist-test-slave-31kt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:14.882079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:15.019465 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:15.019922 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:15.020000 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:15.024746 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:15.024884 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:15.024919 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:15.024946 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:15.024972 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:15.031628 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:15.033448 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:15.034960 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:15.047609 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:15.047575 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:15.047575 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:15.047986 3939 server_base.cc:1034] running on GCE node
I20260925 16:27:15.049033 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:15.050822 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:15.052030 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790353635051994 us; error 63 us; skew 500 ppm
Sep 25 16:27:15 dist-test-slave-31kt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:15.056687 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:15.058379 3939 webserver.cc:460] Webserver started at http://127.0.98.62:37831/ using document root <none> and password file <none>
I20260925 16:27:15.059695 3939 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:15.059819 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:15.060221 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:15.063478 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "4e0073a10f3b49b195ebd6a162afa13b"
format_stamp: "Formatted at 2026-09-25 16:27:15 on dist-test-slave-31kt"
server_key: "b798753bf314482d05b913955a65b5a3"
server_key_iv: "62948a805e4f52362518696a4359306f"
server_key_version: "encryptionkey@0"
I20260925 16:27:15.064343 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e0073a10f3b49b195ebd6a162afa13b"
format_stamp: "Formatted at 2026-09-25 16:27:15 on dist-test-slave-31kt"
server_key: "b798753bf314482d05b913955a65b5a3"
server_key_iv: "62948a805e4f52362518696a4359306f"
server_key_version: "encryptionkey@0"
I20260925 16:27:15.070878 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 16:27:15.075205 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:15.077512 3939 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260925 16:27:15.077800 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "4e0073a10f3b49b195ebd6a162afa13b"
format_stamp: "Formatted at 2026-09-25 16:27:15 on dist-test-slave-31kt"
server_key: "b798753bf314482d05b913955a65b5a3"
server_key_iv: "62948a805e4f52362518696a4359306f"
server_key_version: "encryptionkey@0"
I20260925 16:27:15.078002 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:15.096334 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:15.101536 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:15.101871 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:15.112370 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44883
I20260925 16:27:15.112417 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44883 every 8 connection(s)
I20260925 16:27:15.113718 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:15.117980 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:15.121735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 3939
I20260925 16:27:15.121935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:15.122284 392 external_mini_cluster.cc:1428] Setting key 9db25f11d93e62072f9339bf704f9f89
I20260925 16:27:15.124920 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b: Bootstrap starting.
I20260925 16:27:15.127866 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:15.128773 4007 log.cc:824] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:15.131865 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b: No bootstrap required, opened a new log
Sep 25 16:27:15 dist-test-slave-31kt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:15.136543 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0073a10f3b49b195ebd6a162afa13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260925 16:27:15.136888 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:15.136930 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0073a10f3b49b195ebd6a162afa13b, State: Initialized, Role: FOLLOWER
I20260925 16:27:15.137488 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0073a10f3b49b195ebd6a162afa13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260925 16:27:15.137642 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:15.137717 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:15.137868 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:15.143285 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:15.124328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.98.62:44883)
0925 16:27:15.124971 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:15.124996 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:15.125052 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:27:15.126051 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:15.126096 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:15.126615 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:15.127089 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:15.128199 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:15.129642 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:15.131221 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:15.131864 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:15.132143 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:15.135596 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:15.135605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:15.135633 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:15.135687 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:15.138118 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:15.139488 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:15.139499 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:15.139511 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:15.139741 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:15.140129 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:15.140133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:15.140135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:15.140867 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:15.141101 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:15.141470 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 16:27:15.142007 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":183,"threads_started":1}
I20260925 16:27:15.148262 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0073a10f3b49b195ebd6a162afa13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260925 16:27:15.148905 4007 leader_election.cc:302] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e0073a10f3b49b195ebd6a162afa13b; no voters:
I20260925 16:27:15.149508 4007 leader_election.cc:288] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:15.149667 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:15.150040 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [term 1 LEADER]: Becoming Leader. State: Replica: 4e0073a10f3b49b195ebd6a162afa13b, State: Running, Role: LEADER
I20260925 16:27:15.150578 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0073a10f3b49b195ebd6a162afa13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } }
I20260925 16:27:15.151638 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:15.153452 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e0073a10f3b49b195ebd6a162afa13b. Latest consensus state: current_term: 1 leader_uuid: "4e0073a10f3b49b195ebd6a162afa13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0073a10f3b49b195ebd6a162afa13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } } }
I20260925 16:27:15.153540 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e0073a10f3b49b195ebd6a162afa13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0073a10f3b49b195ebd6a162afa13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44883 } } }
I20260925 16:27:15.153743 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:15.153791 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:15.159307 4027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:15.159453 4027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:15.159570 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:15.160667 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:15.168867 4025 catalog_manager.cc:1347] Generated new cluster ID: a727352e0a8b4fb8851efa72f1046640
I20260925 16:27:15.168998 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:15.189548 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:15.191232 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:15.202450 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e0073a10f3b49b195ebd6a162afa13b: Generated new TSK 0
I20260925 16:27:15.203586 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260925 16:27:16.118561 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.908s user 0.004s sys 0.001s
I20260925 16:27:16.119270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44883
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:16.269562 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:16.270047 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:16.270112 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:16.275012 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:16.275117 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:16.275190 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:16.281929 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:16.283643 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:16.285271 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:16.299259 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:16.299350 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:16.299335 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:16.299585 4033 server_base.cc:1034] running on GCE node
I20260925 16:27:16.300212 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:16.301229 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:16.304634 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790353636304519 us; error 125 us; skew 500 ppm
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:16.308962 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:16.310717 4033 webserver.cc:460] Webserver started at http://127.0.98.1:45989/ using document root <none> and password file <none>
I20260925 16:27:16.311605 4033 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:16.311702 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:16.312198 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:16.315122 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "d14c67f4c0d04f6f8d2067a5fcb08709"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "6a755d77fdda998c9e8979b934ec84b0"
server_key_iv: "5de8da3f9099af821b80366a58c6ee7e"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.315832 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d14c67f4c0d04f6f8d2067a5fcb08709"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "6a755d77fdda998c9e8979b934ec84b0"
server_key_iv: "5de8da3f9099af821b80366a58c6ee7e"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.321223 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:16.326238 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.328284 4033 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260925 16:27:16.328473 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "d14c67f4c0d04f6f8d2067a5fcb08709"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "6a755d77fdda998c9e8979b934ec84b0"
server_key_iv: "5de8da3f9099af821b80366a58c6ee7e"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.328768 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:16.341084 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:16.345461 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:16.345731 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:16.346483 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:16.347507 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:16.347563 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.347618 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:16.347633 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.359117 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36637
I20260925 16:27:16.359198 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36637 every 8 connection(s)
I20260925 16:27:16.360519 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:16.370012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4033
I20260925 16:27:16.370165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:16.370554 392 external_mini_cluster.cc:1428] Setting key 405f775dd7f0b3a6b4a353931ec6ae9a
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:16.378913 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.363523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58733 (local address 127.0.98.62:44883)
0925 16:27:16.363828 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:16.363832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:16.364983 (+ 1151us) server_negotiation.cc:407] Connection header received
0925 16:27:16.366761 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:16.366764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:16.366838 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:16.366930 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:16.368408 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.368977 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.370419 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.370844 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.371024 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.374947 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:16.374951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:16.374953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:16.374986 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:16.376579 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.377211 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.377214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.377215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.377297 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.377660 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.377662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.377664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.377914 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:16.378006 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:16.378569 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 16:27:16.378725 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
I20260925 16:27:16.380152 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.363988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44883 (local address 127.0.98.1:58733)
0925 16:27:16.364755 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:16.364831 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:16.366448 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:16.367176 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:16.367185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:16.367632 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:16.368222 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:16.368237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.369114 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.369117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:16.370207 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:16.370221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.371127 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.371134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:16.371384 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.372283 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:16.372303 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:16.374671 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:16.376748 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:16.376757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:16.376779 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:16.377092 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:16.377381 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:16.377384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:16.377386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:16.377555 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:16.378016 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:16.378024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:16.378307 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 16:27:16.378491 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 16:27:16.378884 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":189,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:27:16.382189 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44883
I20260925 16:27:16.382582 4162 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:16.383401 4162 heartbeater.cc:511] Master 127.0.98.62:44883 requested a full tablet report, sending...
I20260925 16:27:16.385850 3969 ts_manager.cc:194] Registered new tserver with Master: d14c67f4c0d04f6f8d2067a5fcb08709 (127.0.98.1:36637)
I20260925 16:27:16.387194 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:16.415369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44883
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:16.555665 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:16.556068 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:16.556120 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:16.560716 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:16.560829 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:16.560901 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:16.567409 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:16.568919 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:16.570250 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:16.582487 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:16.582571 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:16.582487 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:16.582795 4168 server_base.cc:1034] running on GCE node
I20260925 16:27:16.583552 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:16.584410 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:16.585662 4168 hybrid_clock.cc:648] HybridClock initialized: now 1790353636585633 us; error 44 us; skew 500 ppm
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:16.589095 4168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:16.590281 4168 webserver.cc:460] Webserver started at http://127.0.98.2:41385/ using document root <none> and password file <none>
I20260925 16:27:16.591302 4168 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:16.591398 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:16.591684 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:16.594384 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "676a1ac9f14e482e89223b5745ec6857"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "a8c02ad0583ce6415bde1a742ac20761"
server_key_iv: "e551b0a5936a6885c831da7d774b4b16"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.595003 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "676a1ac9f14e482e89223b5745ec6857"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "a8c02ad0583ce6415bde1a742ac20761"
server_key_iv: "e551b0a5936a6885c831da7d774b4b16"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.599572 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:16.602761 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.604724 4168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:27:16.604897 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "676a1ac9f14e482e89223b5745ec6857"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "a8c02ad0583ce6415bde1a742ac20761"
server_key_iv: "e551b0a5936a6885c831da7d774b4b16"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.605032 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:16.620640 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:16.624756 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:16.624971 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:16.625682 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:16.626698 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:16.626748 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.626806 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:16.626835 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.637050 4168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45961
I20260925 16:27:16.637112 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45961 every 8 connection(s)
I20260925 16:27:16.638578 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:16.644670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4168
I20260925 16:27:16.644802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:16.645064 392 external_mini_cluster.cc:1428] Setting key 82ea00fa7216cc6b71f4305e00e82d4b
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:16.660126 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.641218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58973 (local address 127.0.98.62:44883)
0925 16:27:16.641528 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:16.641533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:16.642799 (+ 1266us) server_negotiation.cc:407] Connection header received
0925 16:27:16.645068 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:16.645073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:16.645181 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:16.645294 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:16.647187 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.648023 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.650159 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.650477 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.650603 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.654430 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:16.654433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:16.654436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:16.654478 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:16.656396 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.657263 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.657270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.657273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.657434 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.658135 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.658140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.658142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.658614 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:16.658744 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:16.659721 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 16:27:16.659939 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 16:27:16.661463 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.641712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44883 (local address 127.0.98.2:58973)
0925 16:27:16.642571 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:16.642639 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:16.644087 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:16.645585 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:16.645595 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:16.646116 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:16.646972 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:16.646998 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.648223 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.648227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:16.649640 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:16.649659 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.650704 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.650709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:16.650952 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.651781 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:16.651806 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:16.654109 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:16.656608 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:16.656617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:16.656641 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:16.657027 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:16.657614 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:16.657617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:16.657618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:16.657941 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:16.658787 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:16.658795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:16.659283 (+ 488us) client_negotiation.cc:769] Sending connection context
0925 16:27:16.659537 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 16:27:16.659985 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":203,"threads_started":1}
I20260925 16:27:16.663322 4297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44883
I20260925 16:27:16.663648 4297 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:16.664351 4297 heartbeater.cc:511] Master 127.0.98.62:44883 requested a full tablet report, sending...
I20260925 16:27:16.665956 3969 ts_manager.cc:194] Registered new tserver with Master: 676a1ac9f14e482e89223b5745ec6857 (127.0.98.2:45961)
I20260925 16:27:16.666863 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:16.678013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44883
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:16.811930 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:16.812273 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:16.812502 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:16.817888 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:16.818003 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:16.818109 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:16.824337 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:16.825651 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:16.826844 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:16.838634 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:16.838788 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:16.838788 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:16.839007 4302 server_base.cc:1034] running on GCE node
I20260925 16:27:16.839550 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:16.840302 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:16.841507 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790353636841465 us; error 66 us; skew 500 ppm
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:16.845148 4302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:16.846585 4302 webserver.cc:460] Webserver started at http://127.0.98.3:38307/ using document root <none> and password file <none>
I20260925 16:27:16.847226 4302 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:16.847303 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:16.847517 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:16.849704 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "40a30a3d0c9e4e53b4b41b4a43218d82"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "6fe288667bef7b79b29605bc2adc69fe"
server_key_iv: "fee848d959595f71c0b2691ec99b4542"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.850348 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40a30a3d0c9e4e53b4b41b4a43218d82"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "6fe288667bef7b79b29605bc2adc69fe"
server_key_iv: "fee848d959595f71c0b2691ec99b4542"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.854946 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:27:16.858265 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.859970 4302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 16:27:16.860163 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "40a30a3d0c9e4e53b4b41b4a43218d82"
format_stamp: "Formatted at 2026-09-25 16:27:16 on dist-test-slave-31kt"
server_key: "6fe288667bef7b79b29605bc2adc69fe"
server_key_iv: "fee848d959595f71c0b2691ec99b4542"
server_key_version: "encryptionkey@0"
I20260925 16:27:16.860298 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:16.881263 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:16.885939 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:16.886178 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:16.886826 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:16.887773 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:16.887823 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.887876 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:16.887905 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:16.898357 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46343
I20260925 16:27:16.898420 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46343 every 8 connection(s)
I20260925 16:27:16.899410 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:16.907316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4302
I20260925 16:27:16.907473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:16.907821 392 external_mini_cluster.cc:1428] Setting key 45c8a24c51c5515398bc2f9600f643d4
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:16.917670 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.901838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49571 (local address 127.0.98.62:44883)
0925 16:27:16.902048 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:16.902054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:16.903240 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 16:27:16.904759 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:16.904763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:16.904833 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:16.904940 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:16.906446 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.907112 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.909448 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.909733 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.909910 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.913242 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:16.913246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:16.913249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:16.913282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:16.914988 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.915754 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.915757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.915759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.915866 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.916332 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.916336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.916337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.916593 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:16.916691 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:16.917344 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 16:27:16.917508 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:27:16.918704 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.902361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44883 (local address 127.0.98.3:49571)
0925 16:27:16.903026 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:16.903084 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:16.904483 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:16.905142 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:16.905151 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:16.905582 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:16.906248 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:16.906266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.907376 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.907380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:16.909276 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:16.909303 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.909986 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.909990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:16.910280 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.911095 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:16.911122 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:16.912994 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:16.915191 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:16.915200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:16.915217 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:16.915605 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:16.916013 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:16.916016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:16.916018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:16.916194 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:16.916735 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:16.916742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:16.917057 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 16:27:16.917266 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 16:27:16.917613 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":173,"threads_started":1}
I20260925 16:27:16.920449 4431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44883
I20260925 16:27:16.920787 4431 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:16.921725 4431 heartbeater.cc:511] Master 127.0.98.62:44883 requested a full tablet report, sending...
I20260925 16:27:16.923278 3971 ts_manager.cc:194] Registered new tserver with Master: 40a30a3d0c9e4e53b4b41b4a43218d82 (127.0.98.3:46343)
I20260925 16:27:16.923939 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49571
I20260925 16:27:16.935833 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:27:16.968051 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.955555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.0.98.1:36637)
0925 16:27:16.956001 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:16.956013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:16.956046 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:27:16.956110 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:16.956117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:16.956402 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:16.956568 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:16.957467 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.958431 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.959407 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.959968 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.960228 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.963451 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:16.963470 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:16.963496 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:16.963544 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:16.965790 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.966479 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.966487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.966488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.966601 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.966919 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.966922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.966923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.967213 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:16.967425 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:16.967694 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 16:27:16.967869 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
Sep 25 16:27:16 dist-test-slave-31kt krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:16.982944 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:16.972045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.0.98.62:44883)
0925 16:27:16.972238 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:16.972242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:16.972338 (+ 96us) server_negotiation.cc:407] Connection header received
0925 16:27:16.972478 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:16.972481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:16.972533 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:16.972614 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:16.973329 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.973932 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.974925 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:16.975184 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:16.975289 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:16.978293 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:16.978298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:16.978300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:16.978341 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:16.980043 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.980664 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.980667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.980669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.980783 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:16.981174 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:16.981178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:16.981180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:16.981592 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:16.981776 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:16.982318 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 16:27:16.982618 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 16:27:16.983579 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46530
I20260925 16:27:16.984588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4033
I20260925 16:27:16.990614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4168
I20260925 16:27:16.996387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4302
I20260925 16:27:17.002945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 3939
2026-09-25T16:27:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3246 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:17 dist-test-slave-31kt krb5kdc[4443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:17 dist-test-slave-31kt krb5kdc[4443](info): set up 2 sockets
Sep 25 16:27:17 dist-test-slave-31kt krb5kdc[4443](info): commencing operation
krb5kdc: starting...
W20260925 16:27:17.970451 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:18 dist-test-slave-31kt krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:18.114929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35477
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:18.269297 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:18.269811 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:18.269896 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:18.274778 4456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:18.274889 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:18.274917 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:18.274940 4456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:18.274961 4456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:18.282742 4456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:18.284384 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:18.285836 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:18.297562 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:18.297547 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:18.297647 4456 server_base.cc:1034] running on GCE node
W20260925 16:27:18.297565 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:18.298684 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:18.300417 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:18.301635 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790353638301606 us; error 55 us; skew 500 ppm
Sep 25 16:27:18 dist-test-slave-31kt krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:18.305861 4456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:18.307125 4456 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:18.307248 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:18.307550 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:18.311076 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "4c2a221a012642cdacd5710b80e84f75"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "85286a9cecdfe5fc588c0c4fc6800129"
server_key_iv: "244dd3d4edb7483652e3fb09d202a0eb"
server_key_version: "encryptionkey@0"
I20260925 16:27:18.311935 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c2a221a012642cdacd5710b80e84f75"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "85286a9cecdfe5fc588c0c4fc6800129"
server_key_iv: "244dd3d4edb7483652e3fb09d202a0eb"
server_key_version: "encryptionkey@0"
I20260925 16:27:18.317621 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:27:18.321812 4470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:18.324029 4456 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:18.324295 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "4c2a221a012642cdacd5710b80e84f75"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "85286a9cecdfe5fc588c0c4fc6800129"
server_key_iv: "244dd3d4edb7483652e3fb09d202a0eb"
server_key_version: "encryptionkey@0"
I20260925 16:27:18.324481 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:18.357059 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:18.362697 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:18.363003 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:18.372977 4456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35477
I20260925 16:27:18.373329 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35477 every 8 connection(s)
I20260925 16:27:18.374012 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:18.375281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4456
I20260925 16:27:18.375413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:18.375741 392 external_mini_cluster.cc:1428] Setting key af0240b6c6f5cfd672a62665ecaa2b03
I20260925 16:27:18.378599 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:18.386979 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: Bootstrap starting.
Sep 25 16:27:18 dist-test-slave-31kt krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:18.390072 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:18.391683 4523 log.cc:824] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:18.395430 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: No bootstrap required, opened a new log
I20260925 16:27:18.395550 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:18.377517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.0.98.62:35477)
0925 16:27:18.378112 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:18.378133 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:18.378182 (+ 49us) server_negotiation.cc:407] Connection header received
0925 16:27:18.379053 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:18.379076 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:18.379554 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:18.379961 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:18.381062 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:18.382381 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:18.383814 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:18.384472 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:18.384654 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:18.388153 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:18.388163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:18.388188 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:18.388242 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:18.390454 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:18.391591 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:18.391603 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:18.391612 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:18.391815 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:18.392340 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:18.392344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:18.392346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:18.393103 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:18.393316 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:18.393954 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 16:27:18.394456 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":232,"threads_started":1}
I20260925 16:27:18.400552 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a221a012642cdacd5710b80e84f75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260925 16:27:18.400986 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:18.401067 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2a221a012642cdacd5710b80e84f75, State: Initialized, Role: FOLLOWER
I20260925 16:27:18.401923 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a221a012642cdacd5710b80e84f75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260925 16:27:18.402243 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:18.402365 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:18.402508 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:18.403738 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a221a012642cdacd5710b80e84f75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260925 16:27:18.404230 4523 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2a221a012642cdacd5710b80e84f75; no voters:
I20260925 16:27:18.404718 4523 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:18.404958 4528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:18.405331 4528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2a221a012642cdacd5710b80e84f75, State: Running, Role: LEADER
I20260925 16:27:18.405821 4528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a221a012642cdacd5710b80e84f75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260925 16:27:18.406389 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:18.408001 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2a221a012642cdacd5710b80e84f75. Latest consensus state: current_term: 1 leader_uuid: "4c2a221a012642cdacd5710b80e84f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a221a012642cdacd5710b80e84f75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } } }
I20260925 16:27:18.407981 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2a221a012642cdacd5710b80e84f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a221a012642cdacd5710b80e84f75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } } }
I20260925 16:27:18.408201 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:18.408201 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75 [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:18.412570 4543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:18.412686 4543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:18.412781 4539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:18.413655 4539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:18.422719 4539 catalog_manager.cc:1347] Generated new cluster ID: 56bb9faf8f0e4a1481a52c1f2731c40b
I20260925 16:27:18.422856 4539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:18.451426 4539 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:18.453737 4539 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:18.465868 4539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: Generated new TSK 0
I20260925 16:27:18.467108 4539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:27:18.524525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:18.664532 4549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:18.664887 4549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:18.664959 4549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:18.671657 4549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:18.671890 4549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:18.672096 4549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:18.680657 4549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:18.682474 4549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:18.684108 4549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:18.698133 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:18.698137 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:18.698133 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:18.698189 4549 server_base.cc:1034] running on GCE node
I20260925 16:27:18.699079 4549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:18.699898 4549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:18.701319 4549 hybrid_clock.cc:648] HybridClock initialized: now 1790353638701284 us; error 51 us; skew 500 ppm
Sep 25 16:27:18 dist-test-slave-31kt krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:18.706621 4549 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:18.707959 4549 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:18.708060 4549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:18.708374 4549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:18.712041 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "03e0e729729c4fb288ddc205bc90237c"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "65895d4af17f0c05d5bb2076dcca3fdb"
server_key_iv: "8e5000124cb51d34c635563f1c94e688"
server_key_version: "encryptionkey@0"
I20260925 16:27:18.713007 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03e0e729729c4fb288ddc205bc90237c"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "65895d4af17f0c05d5bb2076dcca3fdb"
server_key_iv: "8e5000124cb51d34c635563f1c94e688"
server_key_version: "encryptionkey@0"
I20260925 16:27:18.719142 4549 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260925 16:27:18.723222 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:18.726403 4549 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260925 16:27:18.726629 4549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "03e0e729729c4fb288ddc205bc90237c"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "65895d4af17f0c05d5bb2076dcca3fdb"
server_key_iv: "8e5000124cb51d34c635563f1c94e688"
server_key_version: "encryptionkey@0"
I20260925 16:27:18.726867 4549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:18.743186 4549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:18.747953 4549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:18.748167 4549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:18.748837 4549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:18.749913 4549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:18.749966 4549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:18.750023 4549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:18.750052 4549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:18.760511 4549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42797
I20260925 16:27:18.760529 4676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42797 every 8 connection(s)
I20260925 16:27:18.761224 4549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:18.765336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4549
I20260925 16:27:18.765503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:18.765924 392 external_mini_cluster.cc:1428] Setting key 4fa37760db55262fff910a5cf6e015f1
Sep 25 16:27:18 dist-test-slave-31kt krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:27:18.784125 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:18.763869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53587 (local address 127.0.98.62:35477)
0925 16:27:18.764030 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:18.764034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:18.765304 (+ 1270us) server_negotiation.cc:407] Connection header received
0925 16:27:18.767451 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:18.767455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:18.767515 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:18.767636 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:18.770451 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:18.771048 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:18.773076 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:18.773484 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:18.773665 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:18.778298 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:18.778305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:18.778309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:18.778363 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:18.780226 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:18.781211 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:18.781215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:18.781217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:18.781349 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:18.781942 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:18.781948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:18.781950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:18.782347 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:18.782481 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:18.783582 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 16:27:18.783953 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:27:18.785557 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:18.764269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.1:53587)
0925 16:27:18.764990 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:18.765085 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:18.767088 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:18.768003 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:18.768015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:18.768880 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:18.770079 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:18.770119 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:18.771286 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:18.771291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:18.772791 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:18.772813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:18.773882 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:18.773888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:18.774202 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:18.775306 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:18.775332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:18.777933 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:18.780457 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:18.780467 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:18.780494 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:18.780986 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:18.781472 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:18.781475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:18.781476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:18.781699 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:18.782545 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:18.782554 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:18.783102 (+ 548us) client_negotiation.cc:769] Sending connection context
0925 16:27:18.783413 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 16:27:18.784014 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":146,"threads_started":1}
I20260925 16:27:18.787833 4677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35477
I20260925 16:27:18.788225 4677 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:18.788961 4677 heartbeater.cc:511] Master 127.0.98.62:35477 requested a full tablet report, sending...
I20260925 16:27:18.791540 4487 ts_manager.cc:194] Registered new tserver with Master: 03e0e729729c4fb288ddc205bc90237c (127.0.98.1:42797)
I20260925 16:27:18.792935 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:18.809726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:18.947577 4682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:18.947917 4682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:18.947986 4682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:18.952800 4682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:18.952907 4682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:18.952998 4682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:18.964416 4682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:18.966025 4682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:18.967314 4682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:18.983268 4687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:18.983372 4690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:18.983359 4688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:18.984536 4682 server_base.cc:1034] running on GCE node
I20260925 16:27:18.985224 4682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:18.986191 4682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:18.987677 4682 hybrid_clock.cc:648] HybridClock initialized: now 1790353638987650 us; error 77 us; skew 500 ppm
Sep 25 16:27:18 dist-test-slave-31kt krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:18.997339 4682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:18.998528 4682 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:18.998639 4682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:18.999533 4682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:19.004403 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "0061c767b98c45f19bffe4dac39f984a"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "87eaa664cf353eb7a3bbfbee7cab7549"
server_key_iv: "d18f46599e3285f5fc7736ebf2db4bfd"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.005571 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0061c767b98c45f19bffe4dac39f984a"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "87eaa664cf353eb7a3bbfbee7cab7549"
server_key_iv: "d18f46599e3285f5fc7736ebf2db4bfd"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.011838 4682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:19.017591 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.020145 4682 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260925 16:27:19.020342 4682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "0061c767b98c45f19bffe4dac39f984a"
format_stamp: "Formatted at 2026-09-25 16:27:18 on dist-test-slave-31kt"
server_key: "87eaa664cf353eb7a3bbfbee7cab7549"
server_key_iv: "d18f46599e3285f5fc7736ebf2db4bfd"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.020534 4682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:19.038017 4682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:19.043687 4682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:19.044003 4682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:19.045095 4682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:19.046453 4682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:19.046530 4682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.046584 4682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:19.046599 4682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.059201 4682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34773
I20260925 16:27:19.060213 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34773 every 8 connection(s)
I20260925 16:27:19.060351 4682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:19.070330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4682
I20260925 16:27:19.070499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:19.070837 392 external_mini_cluster.cc:1428] Setting key adc08c4ee51f149d8991d1c456815f63
Sep 25 16:27:19 dist-test-slave-31kt krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:27:19.084920 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.063643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44347 (local address 127.0.98.62:35477)
0925 16:27:19.063936 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.063940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.065458 (+ 1518us) server_negotiation.cc:407] Connection header received
0925 16:27:19.067227 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.067231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.067313 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.067402 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:19.069094 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.069664 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.071233 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.071486 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.071610 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.079383 (+ 7773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:19.079388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:19.079391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:19.079436 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:19.082084 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:19.082870 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:19.082874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:19.082875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:19.082991 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:19.083384 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:19.083395 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:19.083397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:19.083615 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:19.083708 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.084349 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.084642 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260925 16:27:19.086129 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.064278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.2:44347)
0925 16:27:19.065184 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.065263 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.066908 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.067650 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.067661 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.068179 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:19.068876 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.068903 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.069859 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.069862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.071072 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.071088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.071685 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.071691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.071927 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.073018 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:19.073043 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:19.078935 (+ 5892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:19.082310 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:19.082319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:19.082339 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:19.082727 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:19.083099 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:19.083102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:19.083103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:19.083273 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:19.083732 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:19.083741 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:19.084072 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.084270 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.084936 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":214,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:19.088222 4810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35477
I20260925 16:27:19.088533 4810 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:19.089123 4810 heartbeater.cc:511] Master 127.0.98.62:35477 requested a full tablet report, sending...
I20260925 16:27:19.090736 4487 ts_manager.cc:194] Registered new tserver with Master: 0061c767b98c45f19bffe4dac39f984a (127.0.98.2:34773)
I20260925 16:27:19.091547 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:19.114511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:19.263315 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:19.263639 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:19.263690 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:19.268242 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:19.268397 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:19.268466 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:19.275058 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:19.276310 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:19.278214 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:19.291796 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:19.291828 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:19.291908 4815 server_base.cc:1034] running on GCE node
W20260925 16:27:19.291821 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:19.292668 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:19.293541 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:19.294763 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790353639294741 us; error 54 us; skew 500 ppm
Sep 25 16:27:19 dist-test-slave-31kt krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:19.298818 4815 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:19.299943 4815 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:19.300026 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:19.300278 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:19.303154 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "17bd4db4ac554434be9f1c0f5733a1f6"
format_stamp: "Formatted at 2026-09-25 16:27:19 on dist-test-slave-31kt"
server_key: "5c6f5b9021322a5a84fa5c37ef7537f8"
server_key_iv: "4b90c68439d547f61bb4ec1095cd7875"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.303890 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "17bd4db4ac554434be9f1c0f5733a1f6"
format_stamp: "Formatted at 2026-09-25 16:27:19 on dist-test-slave-31kt"
server_key: "5c6f5b9021322a5a84fa5c37ef7537f8"
server_key_iv: "4b90c68439d547f61bb4ec1095cd7875"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.309528 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:27:19.313946 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.316031 4815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:19.316231 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "17bd4db4ac554434be9f1c0f5733a1f6"
format_stamp: "Formatted at 2026-09-25 16:27:19 on dist-test-slave-31kt"
server_key: "5c6f5b9021322a5a84fa5c37ef7537f8"
server_key_iv: "4b90c68439d547f61bb4ec1095cd7875"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.316421 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:19.334791 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:19.339905 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:19.340202 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:19.341190 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:19.342468 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:19.342538 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.342590 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:19.342604 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.355079 4815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44753
I20260925 16:27:19.355134 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44753 every 8 connection(s)
I20260925 16:27:19.355937 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:19.366185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4815
I20260925 16:27:19.366331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:19.366815 392 external_mini_cluster.cc:1428] Setting key 764571ba0b180070aed0761dc55f1dd2
Sep 25 16:27:19 dist-test-slave-31kt krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:19.377635 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.358832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44485 (local address 127.0.98.62:35477)
0925 16:27:19.359076 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.359080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.360700 (+ 1620us) server_negotiation.cc:407] Connection header received
0925 16:27:19.362813 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.362818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.362913 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.363002 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:19.365264 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.365910 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.367455 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.367998 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.368182 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.372258 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:19.372263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:19.372265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:19.372305 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:19.374358 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:19.375249 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:19.375254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:19.375256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:19.375366 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:19.376054 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:19.376057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:19.376059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:19.376352 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:19.376465 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.377241 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.377435 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 16:27:19.379061 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.359393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.3:44485)
0925 16:27:19.360225 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.360348 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.362398 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.363421 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.363433 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.364205 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:19.364932 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.364955 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.366081 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.366085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.367249 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.367268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.368294 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.368300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.368533 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.369409 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:19.369434 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:19.371937 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:19.374692 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:19.374701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:19.374722 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:19.375111 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:19.375471 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:19.375474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:19.375476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:19.375883 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:19.376520 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:19.376528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:19.376945 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.377185 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.377600 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":207,"threads_started":1}
I20260925 16:27:19.380995 4943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35477
I20260925 16:27:19.381501 4943 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:19.382206 4943 heartbeater.cc:511] Master 127.0.98.62:35477 requested a full tablet report, sending...
I20260925 16:27:19.383540 4486 ts_manager.cc:194] Registered new tserver with Master: 17bd4db4ac554434be9f1c0f5733a1f6 (127.0.98.3:44753)
I20260925 16:27:19.384212 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44485
I20260925 16:27:19.397209 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:27:19.409680 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.400166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.98.62:35477)
0925 16:27:19.400369 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.400374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.400490 (+ 116us) server_negotiation.cc:407] Connection header received
0925 16:27:19.400658 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.400662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.400723 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.400804 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:19.401715 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.402415 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.403421 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.403667 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.403772 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.405550 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:19.405555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:19.405557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:19.405596 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:19.407502 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:19.408161 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:19.408165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:19.408166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:19.408279 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:19.408550 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:19.408553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:19.408555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:19.408910 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:19.409075 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.409322 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.409483 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 16:27:19.414459 4486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:19.418473 4486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:19.440358 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.432526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34773 (local address 127.0.0.1:42694)
0925 16:27:19.433132 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.433146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.433250 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.434120 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.434124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.434145 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:19.434323 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.434331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.435937 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.435940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.438079 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.438093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.439459 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.439464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.439663 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.439733 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.439870 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.440078 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":384,"thread_start_us":98,"threads_started":1}
I20260925 16:27:19.440644 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.431500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42797 (local address 127.0.0.1:53420)
0925 16:27:19.432512 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.432529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.432665 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.433884 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.433890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.433922 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:19.434190 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.434199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.435876 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.435881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.438138 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.438146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.439459 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.439464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.440211 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.440236 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.440320 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.440424 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":1536,"thread_start_us":111,"threads_started":1}
I20260925 16:27:19.440317 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.431492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44753 (local address 127.0.0.1:51726)
0925 16:27:19.432301 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.432395 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.432617 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.433976 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.433979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.433991 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:19.434192 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.434199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.435892 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.435897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.437670 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.437688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.439389 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.439394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.439845 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.439886 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.439963 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.440103 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":189,"thread_start_us":138,"threads_started":1}
I20260925 16:27:19.441171 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.433127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.0.98.2:34773)
0925 16:27:19.433552 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.433565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.433590 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:27:19.433673 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.433680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.433921 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.434484 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:19.434691 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.435785 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.438288 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.439279 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.440503 (+ 1224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.440739 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.440809 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.440960 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":149,"threads_started":1}
I20260925 16:27:19.441614 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.432590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.0.98.3:44753)
0925 16:27:19.433261 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.433276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.433311 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:27:19.433525 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.433533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.433798 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.434404 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:19.434605 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.435727 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.437937 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.439130 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.441021 (+ 1891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.441254 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.441318 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.441441 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":179,"threads_started":1}
I20260925 16:27:19.442292 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.431762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.0.98.1:42797)
0925 16:27:19.433110 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.433120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.433153 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:27:19.433277 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.433285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.433606 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.433916 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:19.434369 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.435702 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.438308 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.439306 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.439680 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.439889 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.440443 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.442121 (+ 1678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":94,"threads_started":1}
I20260925 16:27:19.445247 4744 tablet_service.cc:1467] Processing CreateTablet for tablet 78425c7564e74b91a12a648cb0783051 (DEFAULT_TABLE table=test-table [id=d0f2cef3120e4155ae13a19897ed2ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:19.445622 4877 tablet_service.cc:1467] Processing CreateTablet for tablet 78425c7564e74b91a12a648cb0783051 (DEFAULT_TABLE table=test-table [id=d0f2cef3120e4155ae13a19897ed2ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:19.445905 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 78425c7564e74b91a12a648cb0783051 (DEFAULT_TABLE table=test-table [id=d0f2cef3120e4155ae13a19897ed2ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:19.447021 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78425c7564e74b91a12a648cb0783051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:19.447018 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78425c7564e74b91a12a648cb0783051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:19.447063 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78425c7564e74b91a12a648cb0783051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:19.456463 4960 tablet_bootstrap.cc:492] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Bootstrap starting.
I20260925 16:27:19.457604 4961 tablet_bootstrap.cc:492] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: Bootstrap starting.
I20260925 16:27:19.459231 4962 tablet_bootstrap.cc:492] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Bootstrap starting.
I20260925 16:27:19.460265 4960 tablet_bootstrap.cc:654] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:19.461058 4961 tablet_bootstrap.cc:654] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:19.461609 4962 tablet_bootstrap.cc:654] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:19.462020 4960 log.cc:824] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:19.462006 4961 log.cc:824] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:19.463268 4962 log.cc:824] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:19.465384 4960 tablet_bootstrap.cc:492] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: No bootstrap required, opened a new log
I20260925 16:27:19.465384 4961 tablet_bootstrap.cc:492] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: No bootstrap required, opened a new log
I20260925 16:27:19.465644 4961 ts_tablet_manager.cc:1397] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:27:19.465644 4960 ts_tablet_manager.cc:1397] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 16:27:19.470283 4960 raft_consensus.cc:348] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.470463 4961 raft_consensus.cc:348] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.470693 4961 raft_consensus.cc:374] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:19.470693 4960 raft_consensus.cc:374] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:19.470746 4960 raft_consensus.cc:729] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bd4db4ac554434be9f1c0f5733a1f6, State: Initialized, Role: FOLLOWER
I20260925 16:27:19.470757 4961 raft_consensus.cc:729] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e0e729729c4fb288ddc205bc90237c, State: Initialized, Role: FOLLOWER
I20260925 16:27:19.470841 4962 tablet_bootstrap.cc:492] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: No bootstrap required, opened a new log
I20260925 16:27:19.471132 4962 ts_tablet_manager.cc:1397] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260925 16:27:19.471371 4961 consensus_queue.cc:260] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.471371 4960 consensus_queue.cc:260] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.472636 4960 ts_tablet_manager.cc:1428] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:19.472636 4961 ts_tablet_manager.cc:1428] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:19.473076 4677 heartbeater.cc:503] Master 127.0.98.62:35477 was elected leader, sending a full tablet report...
I20260925 16:27:19.473076 4943 heartbeater.cc:503] Master 127.0.98.62:35477 was elected leader, sending a full tablet report...
I20260925 16:27:19.476074 4962 raft_consensus.cc:348] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.476491 4962 raft_consensus.cc:374] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:19.476549 4962 raft_consensus.cc:729] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061c767b98c45f19bffe4dac39f984a, State: Initialized, Role: FOLLOWER
I20260925 16:27:19.477157 4962 consensus_queue.cc:260] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.478233 4810 heartbeater.cc:503] Master 127.0.98.62:35477 was elected leader, sending a full tablet report...
I20260925 16:27:19.478704 4962 ts_tablet_manager.cc:1428] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260925 16:27:19.484606 4968 raft_consensus.cc:482] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:19.484880 4968 raft_consensus.cc:504] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.486598 4968 leader_election.cc:288] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17bd4db4ac554434be9f1c0f5733a1f6 (127.0.98.3:44753), 03e0e729729c4fb288ddc205bc90237c (127.0.98.1:42797)
I20260925 16:27:19.491529 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.487114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43779 (local address 127.0.98.1:42797)
0925 16:27:19.487405 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.487411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.487686 (+ 275us) server_negotiation.cc:407] Connection header received
0925 16:27:19.487882 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.487886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.487982 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.488062 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:19.488360 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.489005 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.490482 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.490929 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.491028 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.491102 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.491300 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.491380 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 16:27:19.491542 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.487845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35917 (local address 127.0.98.3:44753)
0925 16:27:19.487984 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.487987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.488002 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:27:19.488061 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.488063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.488123 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.488230 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:19.488516 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.489067 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.490432 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.490908 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.491011 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.491095 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.491272 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.491362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:27:19.491555 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.487104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44753 (local address 127.0.98.2:35917)
0925 16:27:19.487763 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.487776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.487886 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.488261 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.488265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.488284 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:19.488413 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.488418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.489184 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.489188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.490299 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.490310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.491036 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.491039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.491144 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.491171 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.491244 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.491366 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":28,"thread_start_us":262,"threads_started":1}
I20260925 16:27:19.491576 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.487008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42797 (local address 127.0.98.2:43779)
0925 16:27:19.487559 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.487576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:19.487743 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:19.488091 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:19.488094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:19.488112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:19.488248 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.488255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.489153 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.489157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.490346 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:19.490359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.491070 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.491074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:19.491182 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.491205 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:27:19.491276 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 16:27:19.491391 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":85}
I20260925 16:27:19.492530 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78425c7564e74b91a12a648cb0783051" candidate_uuid: "0061c767b98c45f19bffe4dac39f984a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" is_pre_election: true
I20260925 16:27:19.492905 4897 raft_consensus.cc:2454] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0061c767b98c45f19bffe4dac39f984a in term 0.
I20260925 16:27:19.492913 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78425c7564e74b91a12a648cb0783051" candidate_uuid: "0061c767b98c45f19bffe4dac39f984a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e0e729729c4fb288ddc205bc90237c" is_pre_election: true
I20260925 16:27:19.493336 4631 raft_consensus.cc:2454] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0061c767b98c45f19bffe4dac39f984a in term 0.
I20260925 16:27:19.493502 4698 leader_election.cc:302] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061c767b98c45f19bffe4dac39f984a, 17bd4db4ac554434be9f1c0f5733a1f6; no voters:
I20260925 16:27:19.493932 4968 raft_consensus.cc:2781] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:19.494019 4968 raft_consensus.cc:482] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:19.494050 4968 raft_consensus.cc:3037] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:19.495471 4968 raft_consensus.cc:504] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.495910 4968 leader_election.cc:288] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [CANDIDATE]: Term 1 election: Requested vote from peers 17bd4db4ac554434be9f1c0f5733a1f6 (127.0.98.3:44753), 03e0e729729c4fb288ddc205bc90237c (127.0.98.1:42797)
I20260925 16:27:19.496559 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78425c7564e74b91a12a648cb0783051" candidate_uuid: "0061c767b98c45f19bffe4dac39f984a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e0e729729c4fb288ddc205bc90237c"
I20260925 16:27:19.496655 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78425c7564e74b91a12a648cb0783051" candidate_uuid: "0061c767b98c45f19bffe4dac39f984a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bd4db4ac554434be9f1c0f5733a1f6"
I20260925 16:27:19.496728 4631 raft_consensus.cc:3037] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:19.496824 4897 raft_consensus.cc:3037] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:19.498175 4631 raft_consensus.cc:2454] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0061c767b98c45f19bffe4dac39f984a in term 1.
I20260925 16:27:19.498202 4897 raft_consensus.cc:2454] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0061c767b98c45f19bffe4dac39f984a in term 1.
I20260925 16:27:19.498615 4697 leader_election.cc:302] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061c767b98c45f19bffe4dac39f984a, 03e0e729729c4fb288ddc205bc90237c; no voters:
I20260925 16:27:19.498829 4968 raft_consensus.cc:2781] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:19.499337 4968 raft_consensus.cc:686] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 1 LEADER]: Becoming Leader. State: Replica: 0061c767b98c45f19bffe4dac39f984a, State: Running, Role: LEADER
I20260925 16:27:19.499882 4968 consensus_queue.cc:237] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } }
I20260925 16:27:19.504030 4486 catalog_manager.cc:5495] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0061c767b98c45f19bffe4dac39f984a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0061c767b98c45f19bffe4dac39f984a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0061c767b98c45f19bffe4dac39f984a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34773 } health_report { overall_health: HEALTHY } } }
W20260925 16:27:19.513200 4678 tablet.cc:2369] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:19.524544 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.520238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.98.2:34773)
0925 16:27:19.520426 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.520431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.520445 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:27:19.520549 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.520552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.520631 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.520755 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:19.521033 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.521623 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.522617 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:19.522832 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:19.522920 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:19.523292 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:19.523911 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:19.524038 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.524267 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.524349 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 16:27:19.533164 4631 raft_consensus.cc:1261] T 78425c7564e74b91a12a648cb0783051 P 03e0e729729c4fb288ddc205bc90237c [term 1 FOLLOWER]: Refusing update from remote peer 0061c767b98c45f19bffe4dac39f984a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:19.533164 4897 raft_consensus.cc:1261] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 1 FOLLOWER]: Refusing update from remote peer 0061c767b98c45f19bffe4dac39f984a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:19.534211 4970 consensus_queue.cc:1035] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bd4db4ac554434be9f1c0f5733a1f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:19.534477 4968 consensus_queue.cc:1035] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e0e729729c4fb288ddc205bc90237c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:19.546108 4975 mvcc.cc:205] Tried to move back new op lower bound from 7333288507515990016 to 7333288507395108864. Current Snapshot: MvccSnapshot[applied={T|T < 7333288507515990016}]
I20260925 16:27:19.556797 4486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:19.557006 4486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:19.560429 4486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c2a221a012642cdacd5710b80e84f75: Sending DeleteTablet for 3 replicas of tablet 78425c7564e74b91a12a648cb0783051
I20260925 16:27:19.562016 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet 78425c7564e74b91a12a648cb0783051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42694
I20260925 16:27:19.562016 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet 78425c7564e74b91a12a648cb0783051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53420
I20260925 16:27:19.563114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4549
I20260925 16:27:19.563997 4877 tablet_service.cc:1514] Processing DeleteTablet for tablet 78425c7564e74b91a12a648cb0783051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51726
I20260925 16:27:19.565454 4984 tablet_replica.cc:330] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: stopping tablet replica
I20260925 16:27:19.565966 4984 raft_consensus.cc:2229] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:19.566376 4984 raft_consensus.cc:2258] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:19.566871 4983 tablet_replica.cc:330] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: stopping tablet replica
I20260925 16:27:19.567222 4983 raft_consensus.cc:2229] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:19.567665 4983 raft_consensus.cc:2258] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:19.568562 4984 ts_tablet_manager.cc:1905] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:19.569201 4983 ts_tablet_manager.cc:1905] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:19.571913 4984 ts_tablet_manager.cc:1918] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:19.572153 4984 log.cc:1196] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/78425c7564e74b91a12a648cb0783051
I20260925 16:27:19.572228 4983 ts_tablet_manager.cc:1918] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:19.572327 4983 log.cc:1196] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/78425c7564e74b91a12a648cb0783051
I20260925 16:27:19.572671 4983 ts_tablet_manager.cc:1939] T 78425c7564e74b91a12a648cb0783051 P 0061c767b98c45f19bffe4dac39f984a: Deleting consensus metadata
I20260925 16:27:19.572671 4984 ts_tablet_manager.cc:1939] T 78425c7564e74b91a12a648cb0783051 P 17bd4db4ac554434be9f1c0f5733a1f6: Deleting consensus metadata
W20260925 16:27:19.573872 4471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42797 (error 108)
I20260925 16:27:19.574324 4472 catalog_manager.cc:4841] TS 0061c767b98c45f19bffe4dac39f984a (127.0.98.2:34773): tablet 78425c7564e74b91a12a648cb0783051 (table test-table [id=d0f2cef3120e4155ae13a19897ed2ae8]) successfully deleted
I20260925 16:27:19.574566 4472 catalog_manager.cc:4841] TS 17bd4db4ac554434be9f1c0f5733a1f6 (127.0.98.3:44753): tablet 78425c7564e74b91a12a648cb0783051 (table test-table [id=d0f2cef3120e4155ae13a19897ed2ae8]) successfully deleted
I20260925 16:27:19.575069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4682
I20260925 16:27:19.576051 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.575314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42797 (local address 127.0.0.1:53428)
0925 16:27:19.575806 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:19.575903 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
W20260925 16:27:19.576462 4471 catalog_manager.cc:4584] TS 03e0e729729c4fb288ddc205bc90237c (127.0.98.1:42797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78425c7564e74b91a12a648cb0783051: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42797: connect: Connection refused (error 111)
I20260925 16:27:19.583268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4815
I20260925 16:27:19.589366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4456
2026-09-25T16:27:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2587 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T16:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:19Z Disabled control of system clock
I20260925 16:27:19.625087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 16:27:19.763705 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:19.764115 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:19.764160 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:19.768888 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:19.769011 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:19.769025 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:19.769043 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:19.769058 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:19.774889 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:19.776526 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:19.778442 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:19.789654 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:19.789738 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:19.789712 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:19.789999 4989 server_base.cc:1034] running on GCE node
I20260925 16:27:19.790886 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:19.792590 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:19.794059 4989 hybrid_clock.cc:648] HybridClock initialized: now 1790353639793991 us; error 82 us; skew 500 ppm
I20260925 16:27:19.796729 4989 webserver.cc:460] Webserver started at http://127.0.98.62:34465/ using document root <none> and password file <none>
I20260925 16:27:19.797528 4989 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:19.797613 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:19.797888 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:19.800432 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "91955127ab7643b1aad4b55e1c8fff33"
format_stamp: "Formatted at 2026-09-25 16:27:19 on dist-test-slave-31kt"
server_key: "264b6522167017c0be4d9ded312ede3d"
server_key_iv: "d71b79a0904f37ecb6606fa9ec70282a"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.801107 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "91955127ab7643b1aad4b55e1c8fff33"
format_stamp: "Formatted at 2026-09-25 16:27:19 on dist-test-slave-31kt"
server_key: "264b6522167017c0be4d9ded312ede3d"
server_key_iv: "d71b79a0904f37ecb6606fa9ec70282a"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.806205 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:27:19.809851 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:19.811961 4989 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:19.812155 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "91955127ab7643b1aad4b55e1c8fff33"
format_stamp: "Formatted at 2026-09-25 16:27:19 on dist-test-slave-31kt"
server_key: "264b6522167017c0be4d9ded312ede3d"
server_key_iv: "d71b79a0904f37ecb6606fa9ec70282a"
server_key_version: "encryptionkey@0"
I20260925 16:27:19.812300 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:19.818737 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:19.819506 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:19.819700 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:19.827446 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40789
I20260925 16:27:19.827572 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40789 every 8 connection(s)
I20260925 16:27:19.828588 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:19.833228 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:19.833546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 4989
I20260925 16:27:19.833680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:19.834004 392 external_mini_cluster.cc:1428] Setting key 0c614f083c5a3dea9467b7c71b04f417
I20260925 16:27:19.841243 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:19.835772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.98.62:40789)
0925 16:27:19.836845 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:19.836865 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:19.836923 (+ 58us) server_negotiation.cc:407] Connection header received
0925 16:27:19.837907 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:19.837932 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:19.838403 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:19.838795 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:19.839769 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:19.839775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:19.839789 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:19.839800 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:19.839865 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:19.839887 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:19.840040 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:19.840119 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 16:27:19.840418 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":252,"threads_started":1}
I20260925 16:27:19.843156 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: Bootstrap starting.
I20260925 16:27:19.846562 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:19.847543 5056 log.cc:824] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:19.850818 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: No bootstrap required, opened a new log
I20260925 16:27:19.855407 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91955127ab7643b1aad4b55e1c8fff33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260925 16:27:19.855813 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:19.855867 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91955127ab7643b1aad4b55e1c8fff33, State: Initialized, Role: FOLLOWER
I20260925 16:27:19.856490 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91955127ab7643b1aad4b55e1c8fff33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260925 16:27:19.856690 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:19.856789 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:19.856891 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:19.858202 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91955127ab7643b1aad4b55e1c8fff33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260925 16:27:19.858651 5056 leader_election.cc:302] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91955127ab7643b1aad4b55e1c8fff33; no voters:
I20260925 16:27:19.859074 5056 leader_election.cc:288] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:19.859328 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:19.859705 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [term 1 LEADER]: Becoming Leader. State: Replica: 91955127ab7643b1aad4b55e1c8fff33, State: Running, Role: LEADER
I20260925 16:27:19.860131 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91955127ab7643b1aad4b55e1c8fff33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260925 16:27:19.860967 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:19.862548 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91955127ab7643b1aad4b55e1c8fff33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91955127ab7643b1aad4b55e1c8fff33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260925 16:27:19.862708 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:19.862509 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91955127ab7643b1aad4b55e1c8fff33. Latest consensus state: current_term: 1 leader_uuid: "91955127ab7643b1aad4b55e1c8fff33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91955127ab7643b1aad4b55e1c8fff33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260925 16:27:19.862843 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33 [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:19.867770 5076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:19.867883 5076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:19.867972 5070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:19.868844 5070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:19.876817 5070 catalog_manager.cc:1347] Generated new cluster ID: d1ca7b7a026f405bad3fc5ae87abea19
I20260925 16:27:19.876925 5070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:19.900382 5070 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:19.900926 5070 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:19.909814 5070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: Generated new TSK 0
I20260925 16:27:19.910905 5070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:19.916260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--builtin_ntp_servers=127.0.98.20:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 16:27:20.077265 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:20.077651 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:20.077710 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:20.083837 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:20.083964 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:20.084038 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:20.090718 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:20.091943 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:20.093447 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:20.105441 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:20.105456 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:20.105455 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:20.105847 5080 server_base.cc:1034] running on GCE node
I20260925 16:27:20.106397 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:20.107215 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:20.108453 5080 hybrid_clock.cc:648] HybridClock initialized: now 1790353640108431 us; error 56 us; skew 500 ppm
I20260925 16:27:20.110780 5080 webserver.cc:460] Webserver started at http://127.0.98.1:46637/ using document root <none> and password file <none>
I20260925 16:27:20.111425 5080 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:20.111503 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:20.111701 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:20.113982 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "826fd8700298427baab6baa2d0eb0371"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "2b280fd973b51c904996eb91af594226"
server_key_iv: "40e66da9d2b276f2f3bf3f1298ba2dde"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.114573 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "826fd8700298427baab6baa2d0eb0371"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "2b280fd973b51c904996eb91af594226"
server_key_iv: "40e66da9d2b276f2f3bf3f1298ba2dde"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.119279 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:20.122947 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.125257 5080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260925 16:27:20.125447 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "826fd8700298427baab6baa2d0eb0371"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "2b280fd973b51c904996eb91af594226"
server_key_iv: "40e66da9d2b276f2f3bf3f1298ba2dde"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.125590 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:20.131011 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:20.131752 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:20.131953 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:20.132587 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:20.133643 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:20.133693 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.133750 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:20.133803 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.143728 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40171
I20260925 16:27:20.143913 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40171 every 8 connection(s)
I20260925 16:27:20.144845 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:20.146703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5080
I20260925 16:27:20.146831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:20.147091 392 external_mini_cluster.cc:1428] Setting key 010225f3599f36ba63bcc1bb8573680c
I20260925 16:27:20.149677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--builtin_ntp_servers=127.0.98.20:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 16:27:20.153560 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.147526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39659 (local address 127.0.98.62:40789)
0925 16:27:20.147732 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.147738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.149295 (+ 1557us) server_negotiation.cc:407] Connection header received
0925 16:27:20.150972 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.150976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.151036 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.151308 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.152928 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.152934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.152936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.152953 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.152983 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.152994 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.153113 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.153331 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.153396 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:27:20.155053 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.148131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:39659)
0925 16:27:20.149093 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.149160 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.150709 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.151551 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.151561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.151710 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.152469 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.152477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.152549 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.152668 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.153116 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.153125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.153131 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.153343 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.153660 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":165,"threads_started":1}
I20260925 16:27:20.156960 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260925 16:27:20.157369 5208 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:20.157980 5208 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260925 16:27:20.160072 5020 ts_manager.cc:194] Registered new tserver with Master: 826fd8700298427baab6baa2d0eb0371 (127.0.98.1:40171)
W20260925 16:27:20.298262 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:20.298666 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:20.298731 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:20.303562 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:20.303663 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:20.303755 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:20.309295 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:20.310917 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:20.312748 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:20.324944 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:20.325125 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:20.325034 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:20.325347 5211 server_base.cc:1034] running on GCE node
I20260925 16:27:20.326092 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:20.326948 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:20.328161 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790353640328134 us; error 57 us; skew 500 ppm
I20260925 16:27:20.330991 5211 webserver.cc:460] Webserver started at http://127.0.98.2:41575/ using document root <none> and password file <none>
I20260925 16:27:20.331835 5211 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:20.331923 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:20.332185 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:20.335114 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "ccf28f9c16c44b89a93f6e64db42a0a5"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "56d79607c7d306b1f89b242d45163b61"
server_key_iv: "2b2b24df0942c6036d575bab483d814f"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.335850 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ccf28f9c16c44b89a93f6e64db42a0a5"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "56d79607c7d306b1f89b242d45163b61"
server_key_iv: "2b2b24df0942c6036d575bab483d814f"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.340813 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:20.344527 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.346580 5211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:20.346761 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "ccf28f9c16c44b89a93f6e64db42a0a5"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "56d79607c7d306b1f89b242d45163b61"
server_key_iv: "2b2b24df0942c6036d575bab483d814f"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.346920 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:20.353140 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:20.354058 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:20.354259 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:20.354995 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:20.356262 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:20.356316 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.356379 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:20.356403 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.368278 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45531
I20260925 16:27:20.368362 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45531 every 8 connection(s)
I20260925 16:27:20.369663 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:20.378638 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.372568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38895 (local address 127.0.98.62:40789)
0925 16:27:20.372824 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.372829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.374648 (+ 1819us) server_negotiation.cc:407] Connection header received
0925 16:27:20.376136 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.376140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.376215 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.376343 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.377999 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.378003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.378005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.378020 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.378047 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.378057 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.378179 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.378410 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.378475 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 16:27:20.379957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5211
I20260925 16:27:20.380113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:20.380468 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.373191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:38895)
0925 16:27:20.374360 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.374432 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.375839 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.376711 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.376721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.376849 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.377585 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.377587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.377641 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.377734 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.378185 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.378195 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.378201 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.378439 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.378804 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":181,"threads_started":1}
I20260925 16:27:20.380510 392 external_mini_cluster.cc:1428] Setting key 7cfdbc2dedf92c9bd2b10e076f3c114b
I20260925 16:27:20.382795 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260925 16:27:20.383345 5339 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:20.384248 5339 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260925 16:27:20.384317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--builtin_ntp_servers=127.0.98.20:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 16:27:20.386202 5019 ts_manager.cc:194] Registered new tserver with Master: ccf28f9c16c44b89a93f6e64db42a0a5 (127.0.98.2:45531)
W20260925 16:27:20.524261 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:20.524730 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:20.524780 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:20.529515 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:20.529627 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:20.529721 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:20.535574 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:20.537120 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:20.538977 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:20.552258 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:20.552271 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:20.552258 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:20.552948 5342 server_base.cc:1034] running on GCE node
I20260925 16:27:20.553505 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:20.554458 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:20.555676 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790353640555651 us; error 58 us; skew 500 ppm
I20260925 16:27:20.558807 5342 webserver.cc:460] Webserver started at http://127.0.98.3:40795/ using document root <none> and password file <none>
I20260925 16:27:20.559688 5342 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:20.559780 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:20.560040 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:20.563191 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "844f95fa946941a0998eb7e0f0335ab2"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "6759fc89789ac170943dbfd69a185343"
server_key_iv: "3554eb011c1bb620f5f25808375c3b11"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.563920 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "844f95fa946941a0998eb7e0f0335ab2"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "6759fc89789ac170943dbfd69a185343"
server_key_iv: "3554eb011c1bb620f5f25808375c3b11"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.568754 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:27:20.572177 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.574272 5342 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:20.574453 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "844f95fa946941a0998eb7e0f0335ab2"
format_stamp: "Formatted at 2026-09-25 16:27:20 on dist-test-slave-31kt"
server_key: "6759fc89789ac170943dbfd69a185343"
server_key_iv: "3554eb011c1bb620f5f25808375c3b11"
server_key_version: "encryptionkey@0"
I20260925 16:27:20.574669 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:20.580687 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:20.581562 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:20.581822 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:20.582628 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:20.583853 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:20.583911 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.583959 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:20.583973 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:20.595041 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43295
I20260925 16:27:20.595069 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43295 every 8 connection(s)
I20260925 16:27:20.596213 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:20.603724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5342
I20260925 16:27:20.603900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:20.604228 392 external_mini_cluster.cc:1428] Setting key 4d73d6a352b0eb5abe1795fcb0327969
I20260925 16:27:20.604434 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.598755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35047 (local address 127.0.98.62:40789)
0925 16:27:20.598950 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.598954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.600450 (+ 1496us) server_negotiation.cc:407] Connection header received
0925 16:27:20.602134 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.602138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.602213 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.602300 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.603857 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.603861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.603871 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.603891 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.603924 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.603937 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.604071 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.604206 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.604264 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:27:20.606359 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.599185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:35047)
0925 16:27:20.600223 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.600287 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.601842 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.602536 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.602544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.602669 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.603435 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.603437 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.603508 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.603625 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.604044 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.604055 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.604062 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.604273 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.604645 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":162,"threads_started":1}
I20260925 16:27:20.608026 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260925 16:27:20.608362 5470 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:20.608980 5470 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260925 16:27:20.610548 5019 ts_manager.cc:194] Registered new tserver with Master: 844f95fa946941a0998eb7e0f0335ab2 (127.0.98.3:43295)
I20260925 16:27:20.620822 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:27:20.626133 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.624041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:40789)
0925 16:27:20.624219 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.624222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.624278 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:27:20.624432 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.624435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.624482 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.624589 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.625665 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.625669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.625671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.625685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.625712 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.625721 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.625868 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.625949 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.625997 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:27:20.631011 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:20.635076 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:20.652117 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.647847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45531 (local address 127.0.0.1:45644)
0925 16:27:20.648784 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.648880 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.649054 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.650003 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.650010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.650120 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.650845 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.650847 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.650874 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.650966 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.651454 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.651469 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.651476 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.651821 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.651873 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":176,"spinlock_wait_cycles":28288,"thread_start_us":139,"threads_started":1}
I20260925 16:27:20.652745 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.648213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.98.2:45531)
0925 16:27:20.649215 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.649236 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.649271 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:27:20.649356 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.649363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.649685 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.649902 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.651144 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.651152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.651171 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.651184 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.651247 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.651285 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.651420 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.652457 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.652552 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":205,"threads_started":1}
I20260925 16:27:20.652732 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.648807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43295 (local address 127.0.0.1:41272)
0925 16:27:20.649609 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.649624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.649718 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.650483 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.650486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.650500 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.651141 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.651143 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.651153 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.651177 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.651653 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.651657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.651661 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.651734 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.651844 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":117,"threads_started":1}
I20260925 16:27:20.653039 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.647841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.0.1:48254)
0925 16:27:20.648784 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.648872 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.649053 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.651152 (+ 2099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.651156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.651177 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.651705 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.651707 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.651720 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.651746 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.652813 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.652817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.652821 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.652886 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.652932 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"spinlock_wait_cycles":16256,"thread_start_us":93,"threads_started":1}
I20260925 16:27:20.653045 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.649437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.0.98.3:43295)
0925 16:27:20.649947 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.649960 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.649987 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:27:20.650081 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.650086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.650256 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.650399 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.651362 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.651371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.651388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.651407 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.651473 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.651525 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.652002 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.652061 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.652157 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":210,"threads_started":1}
I20260925 16:27:20.653843 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.649335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.98.1:40171)
0925 16:27:20.650054 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.650061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.650093 (+ 32us) server_negotiation.cc:407] Connection header received
0925 16:27:20.650155 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.650161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.650976 (+ 815us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.651147 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.652605 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.652612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.652624 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.652634 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.652676 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.652699 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.653461 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.653517 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.653606 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":131,"threads_started":1}
I20260925 16:27:20.657112 5273 tablet_service.cc:1467] Processing CreateTablet for tablet 9a636dee731f4f56940527ca14d299c5 (DEFAULT_TABLE table=test-table [id=fac115478bd04540aaf44c64cb6cd685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:20.657298 5402 tablet_service.cc:1467] Processing CreateTablet for tablet 9a636dee731f4f56940527ca14d299c5 (DEFAULT_TABLE table=test-table [id=fac115478bd04540aaf44c64cb6cd685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:20.658627 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a636dee731f4f56940527ca14d299c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:20.658877 5402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a636dee731f4f56940527ca14d299c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:20.660200 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 9a636dee731f4f56940527ca14d299c5 (DEFAULT_TABLE table=test-table [id=fac115478bd04540aaf44c64cb6cd685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:20.661656 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a636dee731f4f56940527ca14d299c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:20.668227 5487 tablet_bootstrap.cc:492] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Bootstrap starting.
I20260925 16:27:20.669134 5488 tablet_bootstrap.cc:492] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Bootstrap starting.
I20260925 16:27:20.671571 5489 tablet_bootstrap.cc:492] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371: Bootstrap starting.
I20260925 16:27:20.671836 5488 tablet_bootstrap.cc:654] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:20.671962 5487 tablet_bootstrap.cc:654] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:20.672847 5488 log.cc:824] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:20.673908 5487 log.cc:824] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:20.674713 5489 tablet_bootstrap.cc:654] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:20.675333 5488 tablet_bootstrap.cc:492] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: No bootstrap required, opened a new log
I20260925 16:27:20.675554 5488 ts_tablet_manager.cc:1397] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:27:20.675664 5489 log.cc:824] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:20.676563 5487 tablet_bootstrap.cc:492] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: No bootstrap required, opened a new log
I20260925 16:27:20.676805 5487 ts_tablet_manager.cc:1397] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 16:27:20.678293 5489 tablet_bootstrap.cc:492] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371: No bootstrap required, opened a new log
I20260925 16:27:20.678557 5489 ts_tablet_manager.cc:1397] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:27:20.680435 5488 raft_consensus.cc:348] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.680792 5488 raft_consensus.cc:374] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:20.680859 5488 raft_consensus.cc:729] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844f95fa946941a0998eb7e0f0335ab2, State: Initialized, Role: FOLLOWER
I20260925 16:27:20.681495 5488 consensus_queue.cc:260] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.681456 5487 raft_consensus.cc:348] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.681890 5487 raft_consensus.cc:374] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:20.681959 5487 raft_consensus.cc:729] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf28f9c16c44b89a93f6e64db42a0a5, State: Initialized, Role: FOLLOWER
I20260925 16:27:20.682590 5470 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
I20260925 16:27:20.682587 5487 consensus_queue.cc:260] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.684006 5487 ts_tablet_manager.cc:1428] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260925 16:27:20.683921 5489 raft_consensus.cc:348] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.684276 5489 raft_consensus.cc:374] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:20.684324 5339 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
I20260925 16:27:20.684347 5489 raft_consensus.cc:729] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826fd8700298427baab6baa2d0eb0371, State: Initialized, Role: FOLLOWER
I20260925 16:27:20.684975 5489 consensus_queue.cc:260] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.685083 5488 ts_tablet_manager.cc:1428] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 16:27:20.686059 5208 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
I20260925 16:27:20.686524 5489 ts_tablet_manager.cc:1428] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 16:27:20.811826 5494 raft_consensus.cc:482] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:20.812220 5494 raft_consensus.cc:504] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.813643 5494 leader_election.cc:288] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826fd8700298427baab6baa2d0eb0371 (127.0.98.1:40171), 844f95fa946941a0998eb7e0f0335ab2 (127.0.98.3:43295)
I20260925 16:27:20.816489 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.814207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39951 (local address 127.0.98.1:40171)
0925 16:27:20.814374 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.814380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.814399 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:27:20.814694 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.814698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.814754 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.814848 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.815867 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.815871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.815874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.815889 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.815915 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.815925 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.816037 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.816269 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.816335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:27:20.816489 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.814448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53967 (local address 127.0.98.3:43295)
0925 16:27:20.814611 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.814616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.814784 (+ 168us) server_negotiation.cc:407] Connection header received
0925 16:27:20.814862 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.814864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.814903 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.814977 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.815867 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.815871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.815874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.815888 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.815915 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.815925 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.816037 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.816274 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.816335 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:27:20.816469 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.813979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.98.2:39951)
0925 16:27:20.814237 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.814261 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.814499 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.814903 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.814905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.814919 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.815647 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.815649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.815657 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.815674 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.816076 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.816079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.816085 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.816202 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.816256 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 16:27:20.816481 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.814250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43295 (local address 127.0.98.2:53967)
0925 16:27:20.814686 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.814698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:20.814783 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:20.815007 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:20.815009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:20.815021 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:20.815580 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:20.815582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:20.815597 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:20.815654 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:20.816070 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:20.816074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:20.816079 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:27:20.816228 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 16:27:20.816294 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":34,"thread_start_us":149,"threads_started":1}
I20260925 16:27:20.817565 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a636dee731f4f56940527ca14d299c5" candidate_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826fd8700298427baab6baa2d0eb0371" is_pre_election: true
I20260925 16:27:20.817538 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a636dee731f4f56940527ca14d299c5" candidate_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844f95fa946941a0998eb7e0f0335ab2" is_pre_election: true
I20260925 16:27:20.817972 5424 raft_consensus.cc:2454] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf28f9c16c44b89a93f6e64db42a0a5 in term 0.
I20260925 16:27:20.817972 5162 raft_consensus.cc:2454] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf28f9c16c44b89a93f6e64db42a0a5 in term 0.
I20260925 16:27:20.818611 5227 leader_election.cc:302] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826fd8700298427baab6baa2d0eb0371, ccf28f9c16c44b89a93f6e64db42a0a5; no voters:
I20260925 16:27:20.819096 5494 raft_consensus.cc:2781] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:20.819267 5494 raft_consensus.cc:482] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:20.819332 5494 raft_consensus.cc:3037] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:20.820715 5494 raft_consensus.cc:504] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.821206 5494 leader_election.cc:288] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 826fd8700298427baab6baa2d0eb0371 (127.0.98.1:40171), 844f95fa946941a0998eb7e0f0335ab2 (127.0.98.3:43295)
I20260925 16:27:20.822082 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a636dee731f4f56940527ca14d299c5" candidate_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844f95fa946941a0998eb7e0f0335ab2"
I20260925 16:27:20.822084 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a636dee731f4f56940527ca14d299c5" candidate_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826fd8700298427baab6baa2d0eb0371"
I20260925 16:27:20.822307 5424 raft_consensus.cc:3037] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:20.822307 5162 raft_consensus.cc:3037] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:20.823962 5162 raft_consensus.cc:2454] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf28f9c16c44b89a93f6e64db42a0a5 in term 1.
I20260925 16:27:20.823985 5424 raft_consensus.cc:2454] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf28f9c16c44b89a93f6e64db42a0a5 in term 1.
I20260925 16:27:20.824535 5227 leader_election.cc:302] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 826fd8700298427baab6baa2d0eb0371, ccf28f9c16c44b89a93f6e64db42a0a5; no voters:
I20260925 16:27:20.824849 5494 raft_consensus.cc:2781] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:20.825178 5494 raft_consensus.cc:686] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 1 LEADER]: Becoming Leader. State: Replica: ccf28f9c16c44b89a93f6e64db42a0a5, State: Running, Role: LEADER
I20260925 16:27:20.825729 5494 consensus_queue.cc:237] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260925 16:27:20.830554 5016 catalog_manager.cc:5495] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf28f9c16c44b89a93f6e64db42a0a5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf28f9c16c44b89a93f6e64db42a0a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } health_report { overall_health: UNKNOWN } } }
W20260925 16:27:20.847589 5471 tablet.cc:2369] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:20.854120 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.852108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.0.98.2:45531)
0925 16:27:20.852265 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:20.852269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:20.852289 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:20.852486 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:20.852488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:20.852561 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:20.852671 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:20.853539 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:20.853542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:20.853545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:20.853557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:20.853585 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:20.853594 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:20.853698 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:20.853906 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 16:27:20.853957 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:27:20.863111 5162 raft_consensus.cc:1261] T 9a636dee731f4f56940527ca14d299c5 P 826fd8700298427baab6baa2d0eb0371 [term 1 FOLLOWER]: Refusing update from remote peer ccf28f9c16c44b89a93f6e64db42a0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:20.863304 5424 raft_consensus.cc:1261] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 1 FOLLOWER]: Refusing update from remote peer ccf28f9c16c44b89a93f6e64db42a0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:20.864399 5497 consensus_queue.cc:1035] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826fd8700298427baab6baa2d0eb0371" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:20.864679 5494 consensus_queue.cc:1035] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844f95fa946941a0998eb7e0f0335ab2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:27:20.871693 5340 tablet.cc:2369] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:20.878044 5501 mvcc.cc:205] Tried to move back new op lower bound from 7333288512963661824 to 7333288512826269696. Current Snapshot: MvccSnapshot[applied={T|T < 7333288512963661824}]
I20260925 16:27:20.878327 5502 mvcc.cc:205] Tried to move back new op lower bound from 7333288512963661824 to 7333288512826269696. Current Snapshot: MvccSnapshot[applied={T|T < 7333288512963661824}]
I20260925 16:27:20.887837 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:20.888088 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:20.891510 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91955127ab7643b1aad4b55e1c8fff33: Sending DeleteTablet for 3 replicas of tablet 9a636dee731f4f56940527ca14d299c5
I20260925 16:27:20.892802 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a636dee731f4f56940527ca14d299c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:20 UTC) from {username='slave'} at 127.0.0.1:48254
I20260925 16:27:20.892808 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a636dee731f4f56940527ca14d299c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:20 UTC) from {username='slave'} at 127.0.0.1:45644
I20260925 16:27:20.892943 5402 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a636dee731f4f56940527ca14d299c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:20 UTC) from {username='slave'} at 127.0.0.1:41272
I20260925 16:27:20.893600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5080
I20260925 16:27:20.894909 5510 tablet_replica.cc:330] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: stopping tablet replica
I20260925 16:27:20.895313 5510 raft_consensus.cc:2229] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:20.895851 5510 raft_consensus.cc:2258] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:20.896330 5512 tablet_replica.cc:330] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: stopping tablet replica
I20260925 16:27:20.896654 5512 raft_consensus.cc:2229] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:20.897014 5512 raft_consensus.cc:2258] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:20.898813 5510 ts_tablet_manager.cc:1905] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:20.898897 5512 ts_tablet_manager.cc:1905] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:20.902233 5512 ts_tablet_manager.cc:1918] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:20.902233 5510 ts_tablet_manager.cc:1918] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:20.902364 5512 log.cc:1196] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/9a636dee731f4f56940527ca14d299c5
I20260925 16:27:20.902557 5510 log.cc:1196] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/9a636dee731f4f56940527ca14d299c5
I20260925 16:27:20.902693 5512 ts_tablet_manager.cc:1939] T 9a636dee731f4f56940527ca14d299c5 P 844f95fa946941a0998eb7e0f0335ab2: Deleting consensus metadata
I20260925 16:27:20.903004 5510 ts_tablet_manager.cc:1939] T 9a636dee731f4f56940527ca14d299c5 P ccf28f9c16c44b89a93f6e64db42a0a5: Deleting consensus metadata
W20260925 16:27:20.903842 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40171 (error 108)
I20260925 16:27:20.904462 5004 catalog_manager.cc:4841] TS ccf28f9c16c44b89a93f6e64db42a0a5 (127.0.98.2:45531): tablet 9a636dee731f4f56940527ca14d299c5 (table test-table [id=fac115478bd04540aaf44c64cb6cd685]) successfully deleted
I20260925 16:27:20.904701 5005 catalog_manager.cc:4841] TS 844f95fa946941a0998eb7e0f0335ab2 (127.0.98.3:43295): tablet 9a636dee731f4f56940527ca14d299c5 (table test-table [id=fac115478bd04540aaf44c64cb6cd685]) successfully deleted
I20260925 16:27:20.905292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5211
I20260925 16:27:20.905511 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:20.905137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.0.1:48258)
0925 16:27:20.905312 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:20.905405 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 16:27:20.906164 5005 catalog_manager.cc:4584] TS 826fd8700298427baab6baa2d0eb0371 (127.0.98.1:40171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a636dee731f4f56940527ca14d299c5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40171: connect: Connection refused (error 111)
I20260925 16:27:20.911341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5342
I20260925 16:27:20.917709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 4989
2026-09-25T16:27:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1326 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 16:27:20.940812 392 mini_oidc.cc:152] Starting JWKS server
I20260925 16:27:20.942857 392 webserver.cc:460] Webserver started at https://127.0.0.1:33545/ using document root <none> and password file <none>
I20260925 16:27:20.943033 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:27:20.943305 392 webserver.cc:460] Webserver started at http://127.0.0.1:42387/ using document root <none> and password file <none>
2026-09-25T16:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:20Z Disabled control of system clock
I20260925 16:27:20.956908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43843
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43843
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/testchainca.cert with env {}
W20260925 16:27:21.094255 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:21.094676 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:21.094732 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:21.099591 5519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:21.099714 5519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:21.099730 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:21.099748 5519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:21.099763 5519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:27:21.099781 5519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:27:21.099793 5519 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33545/jwks.json
I20260925 16:27:21.106591 5519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:21.108174 5519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:21.109544 5519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:21.120811 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:21.120811 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:21.121026 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:21.121196 5519 server_base.cc:1034] running on GCE node
I20260925 16:27:21.122098 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:21.123528 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:21.124805 5519 hybrid_clock.cc:648] HybridClock initialized: now 1790353641124767 us; error 67 us; skew 500 ppm
I20260925 16:27:21.127374 5519 webserver.cc:460] Webserver started at http://127.0.98.62:46245/ using document root <none> and password file <none>
I20260925 16:27:21.128086 5519 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:21.128162 5519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:21.128376 5519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:21.130875 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "6624a50201514965ae342be52c10adfb"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "3d2833cb029da59788eb03626eedbf50"
server_key_iv: "2211eadd241ae86c037cffc9b68100f9"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.131482 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "6624a50201514965ae342be52c10adfb"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "3d2833cb029da59788eb03626eedbf50"
server_key_iv: "2211eadd241ae86c037cffc9b68100f9"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.136270 5519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:21.140236 5533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:21.142396 5519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:21.142596 5519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "6624a50201514965ae342be52c10adfb"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "3d2833cb029da59788eb03626eedbf50"
server_key_iv: "2211eadd241ae86c037cffc9b68100f9"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.142756 5519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:21.179631 5519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:21.180485 5519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:21.180696 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:21.189589 5519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43843
I20260925 16:27:21.189625 5585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43843 every 8 connection(s)
I20260925 16:27:21.190801 5519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:21.195500 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:21.197199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5519
I20260925 16:27:21.197336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:21.197639 392 external_mini_cluster.cc:1428] Setting key 170219e128b78fbda2c1294844c7957a
I20260925 16:27:21.204090 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb: Bootstrap starting.
I20260925 16:27:21.208287 5586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:21.209497 5586 log.cc:824] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:21.211464 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:21.199558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.98.62:43843)
0925 16:27:21.200399 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:21.200418 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:21.200467 (+ 49us) server_negotiation.cc:407] Connection header received
0925 16:27:21.201544 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:21.201577 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:21.202321 (+ 744us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:21.202788 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:21.203271 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.204494 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.206224 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.206879 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.207079 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.208352 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:21.208359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:21.208384 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:21.209028 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:21.209177 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:21.209210 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:21.209353 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:21.209867 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 16:27:21.210304 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":163,"threads_started":1}
I20260925 16:27:21.213181 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb: No bootstrap required, opened a new log
I20260925 16:27:21.216005 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:27:21.218719 5586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624a50201514965ae342be52c10adfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 16:27:21.219133 5586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:21.219180 5586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6624a50201514965ae342be52c10adfb, State: Initialized, Role: FOLLOWER
I20260925 16:27:21.219802 5586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624a50201514965ae342be52c10adfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 16:27:21.219972 5586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:21.220048 5586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:21.220211 5586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:21.221659 5586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624a50201514965ae342be52c10adfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 16:27:21.222242 5586 leader_election.cc:302] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6624a50201514965ae342be52c10adfb; no voters:
I20260925 16:27:21.222725 5586 leader_election.cc:288] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:21.223011 5591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:21.223376 5591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [term 1 LEADER]: Becoming Leader. State: Replica: 6624a50201514965ae342be52c10adfb, State: Running, Role: LEADER
I20260925 16:27:21.223879 5591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624a50201514965ae342be52c10adfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 16:27:21.224807 5586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:21.225603 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6624a50201514965ae342be52c10adfb. Latest consensus state: current_term: 1 leader_uuid: "6624a50201514965ae342be52c10adfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624a50201514965ae342be52c10adfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } } }
I20260925 16:27:21.225812 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:21.226029 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6624a50201514965ae342be52c10adfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6624a50201514965ae342be52c10adfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } } }
I20260925 16:27:21.226226 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:21.226292 5596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:21.230136 5596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:21.237737 5596 catalog_manager.cc:1347] Generated new cluster ID: 5a6c81f24617433aa709237029bcbcf9
I20260925 16:27:21.237895 5596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:21.259974 5596 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:21.261402 5596 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:21.271953 5596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6624a50201514965ae342be52c10adfb: Generated new TSK 0
I20260925 16:27:21.272985 5596 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 16:27:21.311767 5616 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 16:27:21.311914 5616 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 16:27:21.312580 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:21.292482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.0.98.62:43843)
0925 16:27:21.292680 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:21.292686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:21.292931 (+ 245us) server_negotiation.cc:407] Connection header received
0925 16:27:21.293217 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:21.293221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:21.293300 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:21.293476 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:27:21.293798 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.294695 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.295824 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.296192 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.296309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.296840 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:27:21.312005 (+ 15165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 16:27:21.312422 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46912: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 16:27:21.312695 5589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46912: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 16:27:21.314549 5615 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:21.292320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43843 (local address 127.0.0.1:46912)
0925 16:27:21.292818 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:21.292833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:21.292997 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:21.293488 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:21.293491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:21.293498 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:27:21.293651 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:21.293656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.294920 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.294923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:21.295673 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:21.295683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.296324 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.296351 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:21.296493 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.296572 (+ 79us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:27:21.312399 (+ 15827us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 16:27:21.312901 (+ 502us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":193,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 16:27:21.315788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5519
2026-09-25T16:27:21Z chronyd exiting
I20260925 16:27:21.354033 392 test_util.cc:175] -----------------------------------------------
I20260925 16:27:21.354143 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790353612433705-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (415 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 16:27:21.355297 392 mini_oidc.cc:152] Starting JWKS server
I20260925 16:27:21.356232 392 webserver.cc:460] Webserver started at https://127.0.0.1:37713/ using document root <none> and password file <none>
I20260925 16:27:21.356346 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:27:21.356596 392 webserver.cc:460] Webserver started at http://127.0.0.1:39027/ using document root <none> and password file <none>
2026-09-25T16:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:21Z Disabled control of system clock
I20260925 16:27:21.366310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32881
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37713/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/testchainca.cert with env {}
W20260925 16:27:21.502370 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:21.502744 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:21.502804 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:21.508893 5622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:21.509052 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:21.509068 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:21.509085 5622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:21.509101 5622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:27:21.509119 5622 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:27:21.509130 5622 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37713/jwks.json
I20260925 16:27:21.516891 5622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37713/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:21.518268 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:21.519411 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:21.529999 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:21.529985 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:21.530145 5622 server_base.cc:1034] running on GCE node
W20260925 16:27:21.529985 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:21.531066 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:21.532389 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:21.533623 5622 hybrid_clock.cc:648] HybridClock initialized: now 1790353641533597 us; error 57 us; skew 500 ppm
I20260925 16:27:21.536159 5622 webserver.cc:460] Webserver started at http://127.0.98.62:43111/ using document root <none> and password file <none>
I20260925 16:27:21.537015 5622 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:21.537106 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:21.537354 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:21.540242 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "e0faae0212444cbf9a0dad114d4450e7"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "9066e71e89f7fa3b6d91d296fa7bf370"
server_key_iv: "4c2323b31d47402c3d12947a96874f68"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.540966 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0faae0212444cbf9a0dad114d4450e7"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "9066e71e89f7fa3b6d91d296fa7bf370"
server_key_iv: "4c2323b31d47402c3d12947a96874f68"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.546095 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:27:21.549706 5636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:21.551718 5622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:21.551954 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "e0faae0212444cbf9a0dad114d4450e7"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "9066e71e89f7fa3b6d91d296fa7bf370"
server_key_iv: "4c2323b31d47402c3d12947a96874f68"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.552105 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:21.567536 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:21.568331 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:21.568526 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:21.576998 5622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32881
I20260925 16:27:21.577054 5688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32881 every 8 connection(s)
I20260925 16:27:21.578151 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:21.582621 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:21.585395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5622
I20260925 16:27:21.585517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:21.585872 392 external_mini_cluster.cc:1428] Setting key ba4ccd34a3ddd01147bbf8bcd051d95a
I20260925 16:27:21.591913 5689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7: Bootstrap starting.
I20260925 16:27:21.595335 5689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:21.596455 5689 log.cc:824] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:21.597602 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:21.587788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.0.98.62:32881)
0925 16:27:21.588568 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:21.588586 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:21.588628 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:27:21.589490 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:21.589510 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:21.590049 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:21.590390 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:21.590963 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.592009 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.593004 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.593539 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.593708 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.594990 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:21.594997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:21.595048 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:21.595658 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:21.595765 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:21.595795 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:21.595925 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:21.596116 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 16:27:21.596519 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":164,"threads_started":1}
I20260925 16:27:21.599467 5689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7: No bootstrap required, opened a new log
I20260925 16:27:21.601956 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:27:21.604426 5689 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0faae0212444cbf9a0dad114d4450e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32881 } }
I20260925 16:27:21.604797 5689 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:21.604873 5689 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0faae0212444cbf9a0dad114d4450e7, State: Initialized, Role: FOLLOWER
I20260925 16:27:21.605454 5689 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0faae0212444cbf9a0dad114d4450e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32881 } }
I20260925 16:27:21.605624 5689 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:21.605697 5689 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:21.605860 5689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:21.607173 5689 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0faae0212444cbf9a0dad114d4450e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32881 } }
I20260925 16:27:21.607631 5689 leader_election.cc:302] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0faae0212444cbf9a0dad114d4450e7; no voters:
I20260925 16:27:21.608073 5689 leader_election.cc:288] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:21.608461 5694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:21.608790 5694 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [term 1 LEADER]: Becoming Leader. State: Replica: e0faae0212444cbf9a0dad114d4450e7, State: Running, Role: LEADER
I20260925 16:27:21.609237 5694 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0faae0212444cbf9a0dad114d4450e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32881 } }
I20260925 16:27:21.609902 5689 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:21.611295 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0faae0212444cbf9a0dad114d4450e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0faae0212444cbf9a0dad114d4450e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32881 } } }
I20260925 16:27:21.611495 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:21.611855 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0faae0212444cbf9a0dad114d4450e7. Latest consensus state: current_term: 1 leader_uuid: "e0faae0212444cbf9a0dad114d4450e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0faae0212444cbf9a0dad114d4450e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32881 } } }
I20260925 16:27:21.611982 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:21.612115 5703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:21.615778 5703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:21.624898 5703 catalog_manager.cc:1347] Generated new cluster ID: ec4d6dd8656b4769be4e92197e010505
I20260925 16:27:21.625037 5703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:21.635023 5703 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:21.636448 5703 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:21.642027 5703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0faae0212444cbf9a0dad114d4450e7: Generated new TSK 0
I20260925 16:27:21.643250 5703 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 16:27:21.670965 5719 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 16:27:21.671098 5719 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 16:27:21.672216 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:21.654969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.0.98.62:32881)
0925 16:27:21.655168 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:21.655174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:21.655500 (+ 326us) server_negotiation.cc:407] Connection header received
0925 16:27:21.655802 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:21.655806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:21.655874 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:21.656025 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:27:21.656289 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.656987 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.658055 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.658309 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.658409 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.658779 (+ 370us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:27:21.671500 (+ 12721us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 16:27:21.672013 (+ 513us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 16:27:21.672329 5692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 16:27:21.672372 5718 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:21.654768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32881 (local address 127.0.0.1:45792)
0925 16:27:21.655281 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:21.655312 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:21.655603 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:21.655994 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:21.655997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:21.656007 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:27:21.656178 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:21.656185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.657130 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.657135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:21.657932 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:21.657942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.658405 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.658438 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:21.658542 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.658576 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:27:21.671902 (+ 13326us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 16:27:21.672158 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":186,"threads_started":1}
I20260925 16:27:21.673415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5622
2026-09-25T16:27:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (358 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 16:27:21.714640 392 mini_oidc.cc:152] Starting JWKS server
I20260925 16:27:21.715668 392 webserver.cc:460] Webserver started at https://127.0.0.1:42359/ using document root <none> and password file <none>
I20260925 16:27:21.715806 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 16:27:21.716133 392 webserver.cc:460] Webserver started at http://127.0.0.1:39883/ using document root <none> and password file <none>
2026-09-25T16:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:21Z Disabled control of system clock
I20260925 16:27:21.727478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40343
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42359/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 16:27:21.868230 5725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:21.868685 5725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:21.868732 5725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:21.873389 5725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:21.873509 5725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:21.873524 5725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:21.873541 5725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:21.873555 5725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:27:21.873571 5725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:27:21.873584 5725 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42359/jwks.json
I20260925 16:27:21.879381 5725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42359/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:21.880973 5725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:21.882846 5725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:21.893733 5730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:21.893855 5731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:21.893855 5733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:21.894384 5725 server_base.cc:1034] running on GCE node
I20260925 16:27:21.895228 5725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:21.896734 5725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:21.897915 5725 hybrid_clock.cc:648] HybridClock initialized: now 1790353641897889 us; error 64 us; skew 500 ppm
I20260925 16:27:21.900610 5725 webserver.cc:460] Webserver started at http://127.0.98.62:38057/ using document root <none> and password file <none>
I20260925 16:27:21.901366 5725 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:21.901444 5725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:21.901670 5725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:21.904454 5725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "c4be9ac4b0584d03aa2a715b5bc09b79"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "306505373b3f1a1e13e04eee8f033d61"
server_key_iv: "9b606e9fb8f67340b6127adc50651de9"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.905186 5725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4be9ac4b0584d03aa2a715b5bc09b79"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "306505373b3f1a1e13e04eee8f033d61"
server_key_iv: "9b606e9fb8f67340b6127adc50651de9"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.910651 5725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:27:21.914857 5739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:21.917327 5725 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:21.917514 5725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "c4be9ac4b0584d03aa2a715b5bc09b79"
format_stamp: "Formatted at 2026-09-25 16:27:21 on dist-test-slave-31kt"
server_key: "306505373b3f1a1e13e04eee8f033d61"
server_key_iv: "9b606e9fb8f67340b6127adc50651de9"
server_key_version: "encryptionkey@0"
I20260925 16:27:21.917728 5725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:21.941380 5725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:21.942560 5725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:21.942818 5725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:21.952807 5725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40343
I20260925 16:27:21.952836 5791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40343 every 8 connection(s)
I20260925 16:27:21.954453 5725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:21.956883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5725
I20260925 16:27:21.957043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:21.957397 392 external_mini_cluster.cc:1428] Setting key 1a4f2f1d1115303439ca64c4a529174b
I20260925 16:27:21.959374 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:21.968425 5792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79: Bootstrap starting.
I20260925 16:27:21.972292 5792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:21.972690 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:21.959605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.0.98.62:40343)
0925 16:27:21.960657 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:21.960682 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:21.960743 (+ 61us) server_negotiation.cc:407] Connection header received
0925 16:27:21.961740 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:21.961802 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:21.962768 (+ 966us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:21.963579 (+ 811us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:21.964247 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.965803 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.967156 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:21.968011 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:21.968246 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:21.969562 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:21.969571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:21.969597 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:21.970510 (+ 913us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:21.970611 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:21.970676 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:21.970827 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:21.971044 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 16:27:21.971391 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":205,"threads_started":1}
I20260925 16:27:21.973318 5792 log.cc:824] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:21.975771 5792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79: No bootstrap required, opened a new log
I20260925 16:27:21.977249 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:27:21.980505 5792 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40343 } }
I20260925 16:27:21.980821 5792 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:21.980859 5792 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4be9ac4b0584d03aa2a715b5bc09b79, State: Initialized, Role: FOLLOWER
I20260925 16:27:21.981331 5792 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40343 } }
I20260925 16:27:21.981460 5792 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:21.981525 5792 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:21.981612 5792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:21.982712 5792 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40343 } }
I20260925 16:27:21.983155 5792 leader_election.cc:302] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4be9ac4b0584d03aa2a715b5bc09b79; no voters:
I20260925 16:27:21.983554 5792 leader_election.cc:288] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:21.983737 5797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:21.984180 5797 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [term 1 LEADER]: Becoming Leader. State: Replica: c4be9ac4b0584d03aa2a715b5bc09b79, State: Running, Role: LEADER
I20260925 16:27:21.984689 5797 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40343 } }
I20260925 16:27:21.985298 5792 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:21.987341 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40343 } } }
I20260925 16:27:21.987567 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:21.988399 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4be9ac4b0584d03aa2a715b5bc09b79. Latest consensus state: current_term: 1 leader_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4be9ac4b0584d03aa2a715b5bc09b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40343 } } }
I20260925 16:27:21.988523 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79 [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:21.991138 5811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:21.991254 5811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:21.991380 5812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:21.992761 5812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:22.002290 5812 catalog_manager.cc:1347] Generated new cluster ID: bc59785c5d4d44ad949b8193f9cffbf7
I20260925 16:27:22.002427 5812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:22.018595 5812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:22.020331 5812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:22.028151 5812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4be9ac4b0584d03aa2a715b5bc09b79: Generated new TSK 0
I20260925 16:27:22.029537 5812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:22.045457 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:22.040650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.0.98.62:40343)
0925 16:27:22.040839 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:22.040844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:22.040862 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:27:22.041006 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:22.041008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:22.041059 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:22.041159 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:27:22.041405 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.042026 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.042945 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.043158 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.043245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:22.043604 (+ 359us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:27:22.045020 (+ 1416us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 16:27:22.045281 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58400: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 16:27:22.045600 5821 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:22.040441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40343 (local address 127.0.0.1:58400)
0925 16:27:22.040730 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:22.040748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:22.040868 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:22.041165 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:22.041168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:22.041176 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:27:22.041303 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:22.041309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.042150 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.042154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:22.042832 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:22.042839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.043243 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.043282 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:22.043366 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:22.043398 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:27:22.045217 (+ 1819us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 16:27:22.045405 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
W20260925 16:27:22.046362 5795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58400: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 16:27:22.046728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5725
E20260925 16:27:22.057487 5822 webserver.cc:533] Webserver: sslize failed:
E20260925 16:27:22.057605 5822 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T16:27:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (372 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T16:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:22Z Disabled control of system clock
I20260925 16:27:22.097119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42185
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/testchainca.cert with env {}
W20260925 16:27:22.234071 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:22.234438 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:22.234488 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:22.238725 5826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:22.238807 5826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:22.238821 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:22.238837 5826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:22.238853 5826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 16:27:22.238869 5826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 16:27:22.238883 5826 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 16:27:22.245277 5826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:22.246544 5826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:22.247622 5826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:22.257651 5832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:22.257651 5834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:22.257654 5831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:22.258114 5826 server_base.cc:1034] running on GCE node
I20260925 16:27:22.258863 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:22.260129 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:22.261353 5826 hybrid_clock.cc:648] HybridClock initialized: now 1790353642261323 us; error 49 us; skew 500 ppm
I20260925 16:27:22.263803 5826 webserver.cc:460] Webserver started at http://127.0.98.62:35861/ using document root <none> and password file <none>
I20260925 16:27:22.264501 5826 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:22.264578 5826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:22.264849 5826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:22.267357 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "83279565146f4a3aa75d75a17b742753"
format_stamp: "Formatted at 2026-09-25 16:27:22 on dist-test-slave-31kt"
server_key: "f811d157548a6f75c0fd6fcffdb277bc"
server_key_iv: "df7a4ccac79a34316afbd5a714122e81"
server_key_version: "encryptionkey@0"
I20260925 16:27:22.268007 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "83279565146f4a3aa75d75a17b742753"
format_stamp: "Formatted at 2026-09-25 16:27:22 on dist-test-slave-31kt"
server_key: "f811d157548a6f75c0fd6fcffdb277bc"
server_key_iv: "df7a4ccac79a34316afbd5a714122e81"
server_key_version: "encryptionkey@0"
I20260925 16:27:22.272615 5826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:27:22.276039 5840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:22.278183 5826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:22.278368 5826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "83279565146f4a3aa75d75a17b742753"
format_stamp: "Formatted at 2026-09-25 16:27:22 on dist-test-slave-31kt"
server_key: "f811d157548a6f75c0fd6fcffdb277bc"
server_key_iv: "df7a4ccac79a34316afbd5a714122e81"
server_key_version: "encryptionkey@0"
I20260925 16:27:22.278507 5826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:22.300390 5826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:22.301257 5826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:22.301476 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:22.310101 5826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42185
I20260925 16:27:22.310129 5892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42185 every 8 connection(s)
I20260925 16:27:22.311215 5826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:22.315585 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:22.315855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5826
I20260925 16:27:22.316001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:22.316323 392 external_mini_cluster.cc:1428] Setting key d23bfb7d7ea0455fead745e5d7985d96
I20260925 16:27:22.325003 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753: Bootstrap starting.
I20260925 16:27:22.328388 5893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:22.329239 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:22.318295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.0.98.62:42185)
0925 16:27:22.319217 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:22.319241 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:22.319296 (+ 55us) server_negotiation.cc:407] Connection header received
0925 16:27:22.320131 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:22.320163 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:22.320828 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:22.321297 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:22.321883 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.323463 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.324606 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.325346 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.325515 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:22.326723 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:22.326731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:22.326751 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:22.327342 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:22.327426 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:22.327457 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:22.327569 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:22.327762 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 16:27:22.328104 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":160,"threads_started":1}
I20260925 16:27:22.329453 5893 log.cc:824] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:22.332230 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753: No bootstrap required, opened a new log
I20260925 16:27:22.333042 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:27:22.336767 5893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83279565146f4a3aa75d75a17b742753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260925 16:27:22.337092 5893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:22.337141 5893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83279565146f4a3aa75d75a17b742753, State: Initialized, Role: FOLLOWER
I20260925 16:27:22.337657 5893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83279565146f4a3aa75d75a17b742753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260925 16:27:22.337831 5893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:22.337906 5893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:22.338018 5893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:22.339243 5893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83279565146f4a3aa75d75a17b742753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260925 16:27:22.339692 5893 leader_election.cc:302] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83279565146f4a3aa75d75a17b742753; no voters:
I20260925 16:27:22.340090 5893 leader_election.cc:288] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:22.340274 5898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:22.340724 5898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [term 1 LEADER]: Becoming Leader. State: Replica: 83279565146f4a3aa75d75a17b742753, State: Running, Role: LEADER
I20260925 16:27:22.341164 5898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83279565146f4a3aa75d75a17b742753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260925 16:27:22.341861 5893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:22.343237 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83279565146f4a3aa75d75a17b742753. Latest consensus state: current_term: 1 leader_uuid: "83279565146f4a3aa75d75a17b742753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83279565146f4a3aa75d75a17b742753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } }
I20260925 16:27:22.343431 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:22.343947 5907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:22.344928 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83279565146f4a3aa75d75a17b742753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83279565146f4a3aa75d75a17b742753" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } }
I20260925 16:27:22.345083 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:22.348289 5907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:22.356060 5907 catalog_manager.cc:1347] Generated new cluster ID: 5b2492a60fe844f09079e080a9f2f783
I20260925 16:27:22.356175 5907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:22.378638 5907 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:22.380471 5907 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:22.388350 5907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83279565146f4a3aa75d75a17b742753: Generated new TSK 0
I20260925 16:27:22.389628 5907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:22.498066 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:22.398722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.0.98.62:42185)
0925 16:27:22.398917 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:22.398922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:22.398948 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:27:22.399138 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:22.399142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:22.399228 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:22.399421 (+ 193us) server_negotiation.cc:226] Negotiated authn=JWT
0925 16:27:22.399758 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.400680 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.401677 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.402094 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.402204 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:22.402654 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 16:27:22.497506 (+ 94852us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 16:27:22.497904 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39444: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 16:27:22.498165 5896 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39444: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 16:27:22.498201 5922 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:22.398542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.0.1:39444)
0925 16:27:22.398836 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:22.398850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:22.398946 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:22.399485 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:22.399489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:22.399498 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 16:27:22.399623 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:22.399628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.400839 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.400843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:22.401539 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:22.401549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:22.402244 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:22.402275 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:22.402369 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:22.402401 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 16:27:22.497793 (+ 95392us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 16:27:22.498033 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":112,"threads_started":1}
I20260925 16:27:22.498911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5826
2026-09-25T16:27:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (431 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (150 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (154 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:22 dist-test-slave-31kt krb5kdc[5934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:22 dist-test-slave-31kt krb5kdc[5934](info): set up 2 sockets
Sep 25 16:27:22 dist-test-slave-31kt krb5kdc[5934](info): commencing operation
krb5kdc: starting...
W20260925 16:27:23.818455 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:23 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:23.958209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:24.098685 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:24.099138 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:24.099186 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:24.104028 5947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:24.104140 5947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:24.104161 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:24.104178 5947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:24.104194 5947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:24.110735 5947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:24.112349 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:24.113785 5947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:24.124863 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:24.124863 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:24.124863 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:24.125041 5947 server_base.cc:1034] running on GCE node
I20260925 16:27:24.126091 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:24.127701 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:24.128901 5947 hybrid_clock.cc:648] HybridClock initialized: now 1790353644128883 us; error 54 us; skew 500 ppm
Sep 25 16:27:24 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:24.137156 5947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:24.138711 5947 webserver.cc:460] Webserver started at http://127.0.98.62:45629/ using document root <none> and password file <none>
I20260925 16:27:24.139509 5947 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:24.139598 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:24.139873 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:24.142639 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "6951802611ca4b11835d5491c568deb2"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "c4afea6a5b86ec1f4c6c8fb61cc12d47"
server_key_iv: "7b97bba82da5c8c8005a16cef89f9fc4"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.143283 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "6951802611ca4b11835d5491c568deb2"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "c4afea6a5b86ec1f4c6c8fb61cc12d47"
server_key_iv: "7b97bba82da5c8c8005a16cef89f9fc4"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.148367 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:24.151659 5962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.153482 5947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 16:27:24.153640 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "6951802611ca4b11835d5491c568deb2"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "c4afea6a5b86ec1f4c6c8fb61cc12d47"
server_key_iv: "7b97bba82da5c8c8005a16cef89f9fc4"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.153874 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:24.173081 5947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:24.177175 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:24.177413 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:24.186813 5947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46825
I20260925 16:27:24.187172 6014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46825 every 8 connection(s)
I20260925 16:27:24.188156 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:24.191643 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:24.197723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 5947
I20260925 16:27:24.197899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:24.198210 392 external_mini_cluster.cc:1428] Setting key ee85c04071acc6356646a59c36eb076d
I20260925 16:27:24.198266 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2: Bootstrap starting.
I20260925 16:27:24.201342 6015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:24.202348 6015 log.cc:824] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:24.204681 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2: No bootstrap required, opened a new log
I20260925 16:27:24.208607 6015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6951802611ca4b11835d5491c568deb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260925 16:27:24.208935 6015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:24.208990 6015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951802611ca4b11835d5491c568deb2, State: Initialized, Role: FOLLOWER
Sep 25 16:27:24 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:24.209559 6015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6951802611ca4b11835d5491c568deb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260925 16:27:24.209805 6015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:24.209897 6015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:24.210032 6015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:24.211148 6015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6951802611ca4b11835d5491c568deb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260925 16:27:24.211580 6015 leader_election.cc:302] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6951802611ca4b11835d5491c568deb2; no voters:
I20260925 16:27:24.211951 6015 leader_election.cc:288] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:24.212148 6020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:24.212460 6020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6951802611ca4b11835d5491c568deb2, State: Running, Role: LEADER
I20260925 16:27:24.212963 6020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6951802611ca4b11835d5491c568deb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260925 16:27:24.213600 6015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:24.215174 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6951802611ca4b11835d5491c568deb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6951802611ca4b11835d5491c568deb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } } }
I20260925 16:27:24.215368 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:24.215776 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6951802611ca4b11835d5491c568deb2. Latest consensus state: current_term: 1 leader_uuid: "6951802611ca4b11835d5491c568deb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6951802611ca4b11835d5491c568deb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } } }
I20260925 16:27:24.215902 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:24.218142 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:24.200168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.0.98.62:46825)
0925 16:27:24.200836 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:24.200859 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:24.200915 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:27:24.202050 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:24.202083 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:24.202661 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:24.203118 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:24.204098 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.205306 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.206237 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.206820 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.207007 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:24.210330 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:24.210340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:24.210368 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:24.210426 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:24.212890 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:24.213654 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:24.213662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:24.213670 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:24.213908 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:24.214272 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:24.214275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:24.214276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:24.214970 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:24.216217 (+ 1247us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:24.216427 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 16:27:24.216847 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":202,"threads_started":1}
W20260925 16:27:24.219304 6035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:24.219405 6035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:24.219491 6032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:24.220501 6032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:24.234539 6032 catalog_manager.cc:1347] Generated new cluster ID: ca677e5aa54f49c1a3efe4642631e754
I20260925 16:27:24.234664 6032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:24.261898 6032 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:24.263451 6032 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:24.282790 6032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6951802611ca4b11835d5491c568deb2: Generated new TSK 0
I20260925 16:27:24.284144 6032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:27:24.317862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46825
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:24.455807 6041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:24.456195 6041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:24.456261 6041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:24.460950 6041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:24.461062 6041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:24.461159 6041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:24.467355 6041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:24.468803 6041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:24.470160 6041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:24.482987 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:24.483042 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:24.482968 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:24.483460 6041 server_base.cc:1034] running on GCE node
I20260925 16:27:24.484040 6041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:24.484834 6041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:24.486106 6041 hybrid_clock.cc:648] HybridClock initialized: now 1790353644486074 us; error 57 us; skew 500 ppm
Sep 25 16:27:24 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:24.489995 6041 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:24.491226 6041 webserver.cc:460] Webserver started at http://127.0.98.1:32985/ using document root <none> and password file <none>
I20260925 16:27:24.491940 6041 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:24.492019 6041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:24.492244 6041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:24.494804 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "652381aca09e428ea50238028986999d"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "4c1af72e084920235c84efd218fdeee7"
server_key_iv: "1aa05c5ba5f3d261e35ab2c5f54a7d25"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.495469 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "652381aca09e428ea50238028986999d"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "4c1af72e084920235c84efd218fdeee7"
server_key_iv: "1aa05c5ba5f3d261e35ab2c5f54a7d25"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.500730 6041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:24.504106 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.506099 6041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:24.506261 6041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "652381aca09e428ea50238028986999d"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "4c1af72e084920235c84efd218fdeee7"
server_key_iv: "1aa05c5ba5f3d261e35ab2c5f54a7d25"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.506387 6041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:24.526312 6041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:24.530534 6041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:24.530735 6041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:24.531448 6041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:24.532472 6041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:24.532526 6041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.532581 6041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:24.532609 6041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.545212 6041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44385
I20260925 16:27:24.545240 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44385 every 8 connection(s)
I20260925 16:27:24.546350 6041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:24.547147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6041
I20260925 16:27:24.547370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:24.547739 392 external_mini_cluster.cc:1428] Setting key 6630dd0422630a0976aec5f832d7c4cd
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 16:27:24 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:24.568367 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:24.549124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55857 (local address 127.0.98.62:46825)
0925 16:27:24.549330 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:24.549337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:24.550918 (+ 1581us) server_negotiation.cc:407] Connection header received
0925 16:27:24.553087 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:24.553092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:24.553181 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:24.553426 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:24.555089 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.555870 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.557526 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.557892 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.557993 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:24.562754 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:24.562760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:24.562763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:24.562811 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:24.564478 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:24.565484 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:24.565488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:24.565489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:24.565602 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:24.566341 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:24.566346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:24.566348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:24.566792 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:24.566973 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:24.568044 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0925 16:27:24.568217 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:27:24.570091 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:24.549863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46825 (local address 127.0.98.1:55857)
0925 16:27:24.550701 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:24.550771 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:24.552717 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:24.553701 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:24.553712 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:24.554239 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:24.554866 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:24.554885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.556107 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.556112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:24.557339 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:24.557359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.558097 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.558102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:24.558360 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:24.559241 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:24.559275 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:24.562383 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:24.564693 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:24.564704 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:24.564734 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:24.565271 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:24.565728 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:24.565732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:24.565735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:24.566070 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:24.566981 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:24.566994 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:24.567549 (+ 555us) client_negotiation.cc:769] Sending connection context
0925 16:27:24.567886 (+ 337us) client_negotiation.cc:240] Negotiation successful
0925 16:27:24.568463 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":229,"threads_started":1}
I20260925 16:27:24.572130 6170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46825
I20260925 16:27:24.572520 6170 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:24.573328 6170 heartbeater.cc:511] Master 127.0.98.62:46825 requested a full tablet report, sending...
I20260925 16:27:24.575641 5979 ts_manager.cc:194] Registered new tserver with Master: 652381aca09e428ea50238028986999d (127.0.98.1:44385)
I20260925 16:27:24.576978 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:24.581578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46825
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:24.724229 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:24.724664 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:24.724717 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:24.729620 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:24.729748 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:24.729883 6175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:24.738224 6175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:24.739897 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:24.741390 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:24.754267 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:24.754509 6175 server_base.cc:1034] running on GCE node
W20260925 16:27:24.754282 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:24.754413 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:24.755231 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:24.756191 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:24.757421 6175 hybrid_clock.cc:648] HybridClock initialized: now 1790353644757399 us; error 50 us; skew 500 ppm
Sep 25 16:27:24 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:24.761168 6175 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:24.762465 6175 webserver.cc:460] Webserver started at http://127.0.98.2:40327/ using document root <none> and password file <none>
I20260925 16:27:24.763135 6175 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:24.763212 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:24.763429 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:24.765919 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba4efdc34e174a088a029ca5c3b33e9e"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "9e5bdc5304b11488e0a4381395cbe5ad"
server_key_iv: "27673464f98f079fb21e8bd1cd8cb1d6"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.766574 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba4efdc34e174a088a029ca5c3b33e9e"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "9e5bdc5304b11488e0a4381395cbe5ad"
server_key_iv: "27673464f98f079fb21e8bd1cd8cb1d6"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.771457 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:24.775087 6190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.777431 6175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:24.777691 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "ba4efdc34e174a088a029ca5c3b33e9e"
format_stamp: "Formatted at 2026-09-25 16:27:24 on dist-test-slave-31kt"
server_key: "9e5bdc5304b11488e0a4381395cbe5ad"
server_key_iv: "27673464f98f079fb21e8bd1cd8cb1d6"
server_key_version: "encryptionkey@0"
I20260925 16:27:24.777982 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:24.799202 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:24.803014 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:24.803223 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:24.803879 6175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:24.805015 6175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:24.805087 6175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.805152 6175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:24.805182 6175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:24.817101 6175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37747
I20260925 16:27:24.817185 6303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37747 every 8 connection(s)
I20260925 16:27:24.818363 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:24.820986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6175
I20260925 16:27:24.821127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:24.821496 392 external_mini_cluster.cc:1428] Setting key b471f6792e9b3ea2ca8e1239bfe1cf87
Sep 25 16:27:24 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:24.838752 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:24.821028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43985 (local address 127.0.98.62:46825)
0925 16:27:24.822267 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:24.822273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:24.822909 (+ 636us) server_negotiation.cc:407] Connection header received
0925 16:27:24.825169 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:24.825174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:24.825378 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:24.825496 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:24.827228 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.827780 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.829151 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.829380 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.829476 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:24.833278 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:24.833283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:24.833285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:24.833324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:24.835393 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:24.836373 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:24.836378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:24.836379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:24.836505 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:24.836987 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:24.836991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:24.836992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:24.837286 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:24.837600 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:24.838329 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 16:27:24.838536 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097}
I20260925 16:27:24.840301 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:24.821694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46825 (local address 127.0.98.2:43985)
0925 16:27:24.822708 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:24.822779 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:24.824731 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:24.825811 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:24.825825 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:24.826335 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:24.827000 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:24.827020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.827938 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.827941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:24.828967 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:24.828990 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:24.829559 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:24.829563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:24.829795 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:24.830652 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:24.830672 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:24.832883 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:24.835615 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:24.835628 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:24.835661 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:24.836165 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:24.836658 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:24.836661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:24.836663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:24.836874 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:24.837516 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:24.837524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:24.837974 (+ 450us) client_negotiation.cc:769] Sending connection context
0925 16:27:24.838216 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 16:27:24.838801 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":206,"threads_started":1}
I20260925 16:27:24.842252 6304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46825
I20260925 16:27:24.842593 6304 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:24.843271 6304 heartbeater.cc:511] Master 127.0.98.62:46825 requested a full tablet report, sending...
I20260925 16:27:24.844851 5979 ts_manager.cc:194] Registered new tserver with Master: ba4efdc34e174a088a029ca5c3b33e9e (127.0.98.2:37747)
I20260925 16:27:24.845657 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:24.855597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46825
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:24.998134 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:24.998582 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:24.998652 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:25.004052 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:25.004173 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:25.004276 6309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:25.012672 6309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:25.014541 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:25.016135 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:25.030913 6315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:25.030990 6309 server_base.cc:1034] running on GCE node
W20260925 16:27:25.030929 6314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:25.031195 6317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:25.032032 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:25.033129 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:25.036638 6309 hybrid_clock.cc:648] HybridClock initialized: now 1790353645036523 us; error 104 us; skew 500 ppm
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.041268 6309 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:25.042840 6309 webserver.cc:460] Webserver started at http://127.0.98.3:39869/ using document root <none> and password file <none>
I20260925 16:27:25.043689 6309 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:25.043790 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:25.044072 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:25.047120 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "917f8239c301409999f3cf73a643b6b1"
format_stamp: "Formatted at 2026-09-25 16:27:25 on dist-test-slave-31kt"
server_key: "d459f0055a6cd6d47c1dcc9c3431ac0a"
server_key_iv: "161c781f0863ad9624362444d5e3655b"
server_key_version: "encryptionkey@0"
I20260925 16:27:25.047968 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "917f8239c301409999f3cf73a643b6b1"
format_stamp: "Formatted at 2026-09-25 16:27:25 on dist-test-slave-31kt"
server_key: "d459f0055a6cd6d47c1dcc9c3431ac0a"
server_key_iv: "161c781f0863ad9624362444d5e3655b"
server_key_version: "encryptionkey@0"
I20260925 16:27:25.053637 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:27:25.058161 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:25.060596 6309 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:25.060856 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "917f8239c301409999f3cf73a643b6b1"
format_stamp: "Formatted at 2026-09-25 16:27:25 on dist-test-slave-31kt"
server_key: "d459f0055a6cd6d47c1dcc9c3431ac0a"
server_key_iv: "161c781f0863ad9624362444d5e3655b"
server_key_version: "encryptionkey@0"
I20260925 16:27:25.061053 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:25.085891 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:25.090360 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:25.090607 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:25.091420 6309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:25.092543 6309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:25.092597 6309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:25.092660 6309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:25.092689 6309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:25.104534 6309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38807
I20260925 16:27:25.104651 6437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38807 every 8 connection(s)
I20260925 16:27:25.105916 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:25.116392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6309
I20260925 16:27:25.116591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:25.116981 392 external_mini_cluster.cc:1428] Setting key fe73da2f7046fcfe5637e6b61e1b8620
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:25.126667 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:25.108815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51459 (local address 127.0.98.62:46825)
0925 16:27:25.109023 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.109027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.110263 (+ 1236us) server_negotiation.cc:407] Connection header received
0925 16:27:25.112264 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.112269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.112357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.112452 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.114389 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.115082 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.116789 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.117116 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.117239 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:25.121253 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:25.121257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:25.121260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:25.121298 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:25.123088 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.123905 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.123908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.123910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.124011 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.124757 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.124760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.124762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.125056 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:25.125154 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:25.126259 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0925 16:27:25.126481 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 16:27:25.127666 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:25.109297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46825 (local address 127.0.98.3:51459)
0925 16:27:25.110017 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:25.110100 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:25.111916 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:25.112722 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:25.112733 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:25.113278 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:25.114173 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:25.114196 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.115244 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.115248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:25.116514 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:25.116532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.117353 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.117359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:25.117619 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:25.118610 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:25.118636 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:25.120996 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:25.123304 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:25.123313 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:25.123338 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:25.123785 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:25.124208 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:25.124213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:25.124215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:25.124577 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:25.125215 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:25.125226 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:25.125789 (+ 563us) client_negotiation.cc:769] Sending connection context
0925 16:27:25.126093 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 16:27:25.126520 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":163,"threads_started":1}
I20260925 16:27:25.129464 6438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46825
I20260925 16:27:25.129810 6438 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:25.130445 6438 heartbeater.cc:511] Master 127.0.98.62:46825 requested a full tablet report, sending...
I20260925 16:27:25.131706 5975 ts_manager.cc:194] Registered new tserver with Master: 917f8239c301409999f3cf73a643b6b1 (127.0.98.3:38807)
I20260925 16:27:25.132397 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51459
I20260925 16:27:25.132922 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/test-admin.keytab.
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.152611 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 16:27:25.156723 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:25.154548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.0.98.1:44385)
0925 16:27:25.154934 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.154944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.154991 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:27:25.155071 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.155078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.155363 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.155549 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.156487 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58008: BlockingRecv error: recv got EOF from 127.0.0.1:58008 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":128,"threads_started":1}
W20260925 16:27:25.157279 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.158779 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.160387 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:27:25.175344 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:25.161895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.0.98.1:44385)
0925 16:27:25.162123 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.162145 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.162223 (+ 78us) server_negotiation.cc:407] Connection header received
0925 16:27:25.162446 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.162449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.162519 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.162623 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.163334 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.164283 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.165360 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.165995 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.166177 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:25.169534 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:25.169555 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:25.169577 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:25.169627 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:25.172505 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.173280 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.173290 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.173291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.173471 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.173966 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.173969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.173970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.174303 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:25.174535 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:25.174926 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 16:27:25.175167 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 16:27:25.180984 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:25.179786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.0.98.1:44385)
0925 16:27:25.179999 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.180003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.180056 (+ 53us) server_negotiation.cc:407] Connection header received
0925 16:27:25.180216 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.180219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.180271 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.180354 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.180857 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58030: BlockingRecv error: recv got EOF from 127.0.0.1:58030 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 16:27:25.181125 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.182634 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.184037 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:27:25.195677 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:25.185341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.0.98.1:44385)
0925 16:27:25.185569 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.185574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.185779 (+ 205us) server_negotiation.cc:407] Connection header received
0925 16:27:25.185955 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.185958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.186007 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.186118 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.186767 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.187302 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.188405 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.188641 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.188739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:25.192225 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:25.192229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:25.192231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:25.192268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:25.193982 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.194473 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.194477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.194479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.194562 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.194819 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.194823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.194825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.195027 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:25.195178 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:25.195389 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:27:25.195515 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 16:27:25.199505 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:25.197952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.0.98.1:44385)
0925 16:27:25.198109 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.198113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.198385 (+ 272us) server_negotiation.cc:407] Connection header received
0925 16:27:25.198501 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.198504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.198564 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.198659 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.199377 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58038: BlockingRecv error: recv got EOF from 127.0.0.1:58038 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 16:27:25.199606 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790353612433705-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.200909 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:25.202188 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:27:25.214490 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:25.203531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.0.98.1:44385)
0925 16:27:25.203823 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:25.203829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:25.203895 (+ 66us) server_negotiation.cc:407] Connection header received
0925 16:27:25.204151 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:25.204155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:25.204228 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:25.204328 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:25.205070 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.205605 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.206689 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:25.206977 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:25.207097 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:25.210325 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:25.210330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:25.210333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:25.210384 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:25.212219 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.212786 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.212789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.212790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.212966 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:25.213324 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:25.213327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:25.213328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:25.213533 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:25.213678 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:25.214147 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 16:27:25.214330 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 16:27:25.216220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6041
I20260925 16:27:25.223106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6175
I20260925 16:27:25.230597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6309
I20260925 16:27:25.238044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 5947
2026-09-25T16:27:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2440 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[6458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[6458](info): set up 2 sockets
Sep 25 16:27:25 dist-test-slave-31kt krb5kdc[6458](info): commencing operation
krb5kdc: starting...
W20260925 16:27:26.222664 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:26 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 16:27:26.374548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:26.530019 6471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:26.530500 6471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:26.530553 6471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:26.535571 6471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:26.535681 6471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:26.535701 6471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:26.535718 6471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:26.535738 6471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:26.542793 6471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:26.544636 6471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:26.546208 6471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:26.558929 6479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:26.558929 6477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:26.558975 6476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:26.559412 6471 server_base.cc:1034] running on GCE node
I20260925 16:27:26.560297 6471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:26.562057 6471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:26.563278 6471 hybrid_clock.cc:648] HybridClock initialized: now 1790353646563257 us; error 47 us; skew 500 ppm
Sep 25 16:27:26 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:26.567236 6471 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 16:27:26.568886 6471 webserver.cc:460] Webserver started at http://127.0.98.62:40703/ using document root <none> and password file <none>
I20260925 16:27:26.569725 6471 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:26.569885 6471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:26.570174 6471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:26.573172 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "538581326ec446cfa4f4f3373754f1fb"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0a75b51d8cb8d1a7234c0c1917483800"
server_key_iv: "3d2d63eb31ad29401faf335eb924ab5b"
server_key_version: "encryptionkey@0"
I20260925 16:27:26.574183 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "538581326ec446cfa4f4f3373754f1fb"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0a75b51d8cb8d1a7234c0c1917483800"
server_key_iv: "3d2d63eb31ad29401faf335eb924ab5b"
server_key_version: "encryptionkey@0"
I20260925 16:27:26.579852 6471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:26.584242 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:26.587054 6471 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 16:27:26.587349 6471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "538581326ec446cfa4f4f3373754f1fb"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0a75b51d8cb8d1a7234c0c1917483800"
server_key_iv: "3d2d63eb31ad29401faf335eb924ab5b"
server_key_version: "encryptionkey@0"
I20260925 16:27:26.587553 6471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:26.612392 6471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:26.616896 6471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:26.617153 6471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:26.626796 6471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34097
I20260925 16:27:26.627173 6538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34097 every 8 connection(s)
I20260925 16:27:26.628279 6471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:26.632067 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:26.635412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6471
I20260925 16:27:26.635589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:26.635926 392 external_mini_cluster.cc:1428] Setting key 205f9f37a692fb8d096626333d62122a
I20260925 16:27:26.639381 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Bootstrap starting.
I20260925 16:27:26.642668 6539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:26.643570 6539 log.cc:824] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:26.646015 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: No bootstrap required, opened a new log
Sep 25 16:27:26 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:26.651348 6539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260925 16:27:26.651712 6539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:26.651810 6539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538581326ec446cfa4f4f3373754f1fb, State: Initialized, Role: FOLLOWER
I20260925 16:27:26.652464 6539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260925 16:27:26.652671 6539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:26.652753 6539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:26.652881 6539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:26.655055 6539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260925 16:27:26.655586 6539 leader_election.cc:302] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538581326ec446cfa4f4f3373754f1fb; no voters:
I20260925 16:27:26.656023 6539 leader_election.cc:288] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:26.656126 6544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:26.656486 6544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 LEADER]: Becoming Leader. State: Replica: 538581326ec446cfa4f4f3373754f1fb, State: Running, Role: LEADER
I20260925 16:27:26.656941 6544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260925 16:27:26.657876 6539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:26.658250 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:26.638225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.0.98.62:34097)
0925 16:27:26.638859 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:26.638881 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:26.638941 (+ 60us) server_negotiation.cc:407] Connection header received
0925 16:27:26.640087 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:26.640118 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:26.640696 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:26.641244 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:26.642354 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:26.643934 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:26.645135 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:26.646126 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:26.646377 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:26.650213 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:26.650223 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:26.650253 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:26.650321 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:26.652760 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:26.653598 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:26.653610 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:26.653622 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:26.653946 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:26.654495 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:26.654500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:26.654502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:26.655327 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:26.655584 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:26.656331 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 16:27:26.656905 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":170,"threads_started":1}
I20260925 16:27:26.659332 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } }
I20260925 16:27:26.659559 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:26.660240 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538581326ec446cfa4f4f3373754f1fb. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } }
I20260925 16:27:26.660446 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:26.665084 6559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:26.665263 6559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:26.665380 6555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:26.666507 6555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:26.675427 6555 catalog_manager.cc:1347] Generated new cluster ID: 3c397eaee7914c7eb7a551bd81a560ba
I20260925 16:27:26.675575 6555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:26.716704 6555 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:26.718339 6555 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:26.729985 6555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Generated new TSK 0
I20260925 16:27:26.731117 6555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 16:27:26.770562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:26.918576 6565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:26.918987 6565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:26.919044 6565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:26.923725 6565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:26.923849 6565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:26.923930 6565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:26.930557 6565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:26.932155 6565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:26.933529 6565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:26.946846 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:26.946844 6573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:26.946856 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:26.947190 6565 server_base.cc:1034] running on GCE node
I20260925 16:27:26.947935 6565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:26.948848 6565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:26.950081 6565 hybrid_clock.cc:648] HybridClock initialized: now 1790353646950059 us; error 47 us; skew 500 ppm
Sep 25 16:27:26 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:26.953840 6565 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:27:26.955266 6565 webserver.cc:460] Webserver started at http://127.0.98.1:39171/ using document root <none> and password file <none>
I20260925 16:27:26.956056 6565 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:26.956151 6565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:26.956421 6565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:26.959604 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "30cce7aad35d402096abc60e5d7a4563"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0e7b5e53513c8105589d0c3eeb1e572a"
server_key_iv: "852ac1242ce8c5509242b4c447d331a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:26.960421 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30cce7aad35d402096abc60e5d7a4563"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0e7b5e53513c8105589d0c3eeb1e572a"
server_key_iv: "852ac1242ce8c5509242b4c447d331a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:26.966257 6565 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 16:27:26.970386 6580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:26.972560 6565 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:27:26.972806 6565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "30cce7aad35d402096abc60e5d7a4563"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0e7b5e53513c8105589d0c3eeb1e572a"
server_key_iv: "852ac1242ce8c5509242b4c447d331a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:26.972993 6565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:27.000638 6565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:27.004940 6565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:27.005179 6565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:27.005985 6565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:27.007130 6565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:27.007187 6565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.007257 6565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:27.007283 6565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.020151 6565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34861
I20260925 16:27:27.020174 6693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34861 every 8 connection(s)
I20260925 16:27:27.021579 6565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:27.031486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6565
I20260925 16:27:27.031646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:27.031957 392 external_mini_cluster.cc:1428] Setting key 245174797b16ab2f72b72614c1347d00
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:27:27.043220 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.024561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58753 (local address 127.0.98.62:34097)
0925 16:27:27.024978 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:27.024983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:27.026234 (+ 1251us) server_negotiation.cc:407] Connection header received
0925 16:27:27.028293 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:27.028299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:27.028396 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:27.028488 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:27.030889 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.031593 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.033677 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.034024 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.034146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.037860 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:27.037865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:27.037868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:27.037915 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:27.039900 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:27.040827 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:27.040832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:27.040834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:27.040970 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:27.041519 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:27.041522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:27.041524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:27.041864 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:27.041994 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:27.042889 (+ 895us) server_negotiation.cc:299] Negotiation successful
0925 16:27:27.043066 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 16:27:27.044600 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.025092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:58753)
0925 16:27:27.025965 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:27.026046 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:27.027852 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:27.028798 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:27.028811 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:27.029641 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:27.030601 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.030626 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.031859 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.031864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.033434 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.033454 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.034238 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.034242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.034457 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.035263 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:27.035287 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:27.037495 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:27.040168 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.040177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.040203 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.040659 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:27.041133 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.041137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.041139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.041382 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:27.042031 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:27.042040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:27.042500 (+ 460us) client_negotiation.cc:769] Sending connection context
0925 16:27:27.042800 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 16:27:27.043308 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":215,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 16:27:27.046722 6694 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260925 16:27:27.047118 6694 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:27.048152 6694 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260925 16:27:27.051033 6503 ts_manager.cc:194] Registered new tserver with Master: 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861)
I20260925 16:27:27.052815 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 16:27:27.070582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:27.222227 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:27.222702 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:27.222795 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:27.228420 6699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:27.228709 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:27.228912 6699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:27.237916 6699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:27.239694 6699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:27.241973 6699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:27.256984 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:27.256984 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:27.257030 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:27.258725 6699 server_base.cc:1034] running on GCE node
I20260925 16:27:27.259498 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:27.260936 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:27.262368 6699 hybrid_clock.cc:648] HybridClock initialized: now 1790353647262328 us; error 56 us; skew 500 ppm
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:27.267154 6699 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:27:27.268496 6699 webserver.cc:460] Webserver started at http://127.0.98.2:39979/ using document root <none> and password file <none>
I20260925 16:27:27.269222 6699 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:27.269354 6699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:27.269600 6699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:27.272269 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "d10d58d5c2f2421bae4e7d593b9776fa"
server_key_iv: "930dc5156c2c607a99ac97e80f291a18"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.272979 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "d10d58d5c2f2421bae4e7d593b9776fa"
server_key_iv: "930dc5156c2c607a99ac97e80f291a18"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.279961 6699 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:27.284663 6714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.287184 6699 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 16:27:27.287456 6699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "d10d58d5c2f2421bae4e7d593b9776fa"
server_key_iv: "930dc5156c2c607a99ac97e80f291a18"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.287643 6699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:27.313160 6699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:27.318684 6699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:27.318956 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:27.319815 6699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:27.321177 6699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:27.321257 6699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.321331 6699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:27.321357 6699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.335348 6699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39793
I20260925 16:27:27.335503 6827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39793 every 8 connection(s)
I20260925 16:27:27.336838 6699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:27.341360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6699
I20260925 16:27:27.341566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:27.342147 392 external_mini_cluster.cc:1428] Setting key fb2772ffe8d868318464577311bd5cd0
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:27.360482 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.339715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36739 (local address 127.0.98.62:34097)
0925 16:27:27.339971 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:27.339977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:27.341627 (+ 1650us) server_negotiation.cc:407] Connection header received
0925 16:27:27.343858 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:27.343862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:27.343937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:27.344011 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:27.346607 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.347456 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.349512 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.349925 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.350057 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.354538 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:27.354551 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:27.354559 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:27.354608 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:27.356278 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:27.357114 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:27.357117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:27.357118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:27.357217 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:27.357743 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:27.357746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:27.357748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:27.358346 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:27.358599 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:27.360100 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0925 16:27:27.360328 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 16:27:27.362282 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.340093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:36739)
0925 16:27:27.341231 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:27.341361 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:27.343511 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:27.344336 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:27.344351 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:27.345080 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:27.346221 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.346254 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.347751 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.347755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.349221 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.349248 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.350189 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.350195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.350478 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.351500 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:27.351527 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:27.353978 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:27.356518 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.356527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.356553 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.356977 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:27.357319 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.357323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.357326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.357523 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:27.358627 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:27.358635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:27.359257 (+ 622us) client_negotiation.cc:769] Sending connection context
0925 16:27:27.359828 (+ 571us) client_negotiation.cc:240] Negotiation successful
0925 16:27:27.360446 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":224,"threads_started":1}
I20260925 16:27:27.364401 6828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260925 16:27:27.364884 6828 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:27.366021 6828 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260925 16:27:27.369048 6503 ts_manager.cc:194] Registered new tserver with Master: fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793)
I20260925 16:27:27.369976 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 16:27:27.383046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:27.560225 6833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:27.560575 6833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:27.560621 6833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:27.566462 6833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:27.566597 6833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:27.566681 6833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:27.574360 6833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:27.576005 6833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:27.577482 6833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:27.591090 6841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:27.591063 6839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:27.591434 6833 server_base.cc:1034] running on GCE node
W20260925 16:27:27.591063 6838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:27.592170 6833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:27.593564 6833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:27.595055 6833 hybrid_clock.cc:648] HybridClock initialized: now 1790353647594988 us; error 107 us; skew 500 ppm
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:27.599236 6833 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:27:27.601369 6833 webserver.cc:460] Webserver started at http://127.0.98.3:33437/ using document root <none> and password file <none>
I20260925 16:27:27.602330 6833 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:27.602452 6833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:27.602730 6833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:27.605365 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "a1991305fb884eaf93a8ec27104cada9"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "f653379e5e00bac506a099af3a40cde8"
server_key_iv: "f2881e7cd1ceabe84c3d9e88c73c589c"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.606089 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a1991305fb884eaf93a8ec27104cada9"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "f653379e5e00bac506a099af3a40cde8"
server_key_iv: "f2881e7cd1ceabe84c3d9e88c73c589c"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.611390 6833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 16:27:27.615144 6848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.617554 6833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:27:27.617878 6833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "a1991305fb884eaf93a8ec27104cada9"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "f653379e5e00bac506a099af3a40cde8"
server_key_iv: "f2881e7cd1ceabe84c3d9e88c73c589c"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.618037 6833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:27.632885 6833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:27.637403 6833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:27.637676 6833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:27.638700 6833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:27.640574 6833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:27.640662 6833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.640736 6833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:27.640766 6833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.652267 6833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40925
I20260925 16:27:27.652433 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40925 every 8 connection(s)
I20260925 16:27:27.653432 6833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:27.655767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6833
I20260925 16:27:27.655908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:27.656304 392 external_mini_cluster.cc:1428] Setting key dc791db4742a90ef2c8ab385106ae7c2
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:27.673873 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.656337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49719 (local address 127.0.98.62:34097)
0925 16:27:27.656557 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:27.656563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:27.658138 (+ 1575us) server_negotiation.cc:407] Connection header received
0925 16:27:27.659993 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:27.659997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:27.660069 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:27.660177 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:27.662159 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.662772 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.664380 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.664675 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.664829 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.668696 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:27.668701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:27.668704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:27.668759 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:27.670676 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:27.671454 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:27.671457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:27.671458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:27.671548 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:27.671928 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:27.671931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:27.671932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:27.672370 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:27.672516 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:27.673361 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 16:27:27.673591 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:27:27.675082 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.657033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:49719)
0925 16:27:27.657923 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:27.658019 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:27.659710 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:27.660500 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:27.660511 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:27.661076 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:27.661914 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.661939 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.662933 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.662936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.664100 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.664116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.664940 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.664944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.665194 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.666110 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:27.666135 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:27.668306 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:27.670898 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.670907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.670929 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.671315 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:27.671642 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.671646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.671648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.671817 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:27.672587 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:27.672597 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:27.673018 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 16:27:27.673266 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 16:27:27.673835 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":271,"threads_started":1}
I20260925 16:27:27.676913 6962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260925 16:27:27.677387 6962 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:27.678216 6962 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260925 16:27:27.679975 6503 ts_manager.cc:194] Registered new tserver with Master: a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925)
I20260925 16:27:27.680740 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49719
I20260925 16:27:27.684820 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 16:27:27.721109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37997
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34097,127.0.98.61:37997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:27.871361 6967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:27.871816 6967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:27.871884 6967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:27.876735 6967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:27.876863 6967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:27.876902 6967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:27.876925 6967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:27.876948 6967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:27.884074 6967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34097,127.0.98.61:37997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
W20260925 16:27:27.884508 6967 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34097,127.0.98.61:37997'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 16:27:27.885928 6967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:27.887421 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:27.899909 6975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:27.899941 6973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:27.899941 6972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:27.900249 6967 server_base.cc:1034] running on GCE node
I20260925 16:27:27.901250 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:27.902846 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:27.904093 6967 hybrid_clock.cc:648] HybridClock initialized: now 1790353647904068 us; error 65 us; skew 500 ppm
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:27.908241 6967 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 16:27:27.909834 6967 webserver.cc:460] Webserver started at http://127.0.98.61:38241/ using document root <none> and password file <none>
I20260925 16:27:27.910641 6967 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:27.910728 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:27.910995 6967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:27.914402 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/instance:
uuid: "bb3a70b12add4fa6b8e6351fff12935c"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "6eeeae7d5106b4ff4f65820bae7813ee"
server_key_iv: "85b55c3eda2205045ac799488e1e37b9"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.915230 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal/instance:
uuid: "bb3a70b12add4fa6b8e6351fff12935c"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "6eeeae7d5106b4ff4f65820bae7813ee"
server_key_iv: "85b55c3eda2205045ac799488e1e37b9"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.920531 6967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:27.924341 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:27.926658 6967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:27:27.926923 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
uuid: "bb3a70b12add4fa6b8e6351fff12935c"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "6eeeae7d5106b4ff4f65820bae7813ee"
server_key_iv: "85b55c3eda2205045ac799488e1e37b9"
server_key_version: "encryptionkey@0"
I20260925 16:27:27.927132 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:27.959638 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:27.963678 6967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:27.963887 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:27.972765 6967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37997
I20260925 16:27:27.973073 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37997 every 8 connection(s)
I20260925 16:27:27.974372 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:27:27.978766 7035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:27.983109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 6967
I20260925 16:27:27.985430 7035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:27:27 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:27:28.002684 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.988519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.98.62:34097)
0925 16:27:27.988686 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:27.988691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:27.989467 (+ 776us) server_negotiation.cc:407] Connection header received
0925 16:27:27.989809 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:27.989813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:27.989881 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:27.989963 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:27.991085 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.991725 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.993562 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.993961 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.994039 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.997365 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:27.997370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:27.997374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:27.997421 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:27.999279 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.000103 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.000108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.000110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.000226 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.000859 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.000863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.000865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.001217 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.001303 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.002244 (+ 941us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.002461 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:27:28.003825 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.986140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.0.98.61:37997)
0925 16:27:27.986998 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:27.987024 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:27.987094 (+ 70us) server_negotiation.cc:407] Connection header received
0925 16:27:27.988148 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:27.988175 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:27.988754 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:27.989176 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:27.990266 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.991586 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.992715 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.993882 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.994084 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.997481 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:27.997490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:27.997508 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:27.997537 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:27.999789 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.000498 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.000508 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.000510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.000747 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.001209 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.001214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.001216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.001964 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.002227 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.002565 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.002711 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":614,"spinlock_wait_cycles":30336,"thread_start_us":332,"threads_started":1}
I20260925 16:27:28.003825 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:27.988410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:50708)
0925 16:27:27.989193 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:27.989314 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:27.989620 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:27.990052 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:27.990060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:27.990631 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:27.990856 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.990872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.991890 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.991893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.993361 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:27.993373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:27.994294 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:27.994298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:27.994383 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:27.995032 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:27.995051 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:27.997087 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:27.999481 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:27.999494 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:27.999510 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:27.999969 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.000401 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.000404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.000406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.000740 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.001420 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.001430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.001963 (+ 533us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.002174 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.002585 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":144,"spinlock_wait_cycles":8576,"thread_start_us":241,"threads_started":1}
I20260925 16:27:28.006032 7035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:27:28.018142 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.006897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37290)
0925 16:27:28.007066 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.007085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.007179 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.007560 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.007563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.007784 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.007923 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.007931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.008999 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.009002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.009638 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.009648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.010336 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.010339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.010431 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.011252 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.011265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.012917 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.016167 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.016171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.016173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.016408 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.016712 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.016715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.016717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.016806 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.017390 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.017395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.017567 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.017731 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.017977 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 16:27:28.018308 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.007030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.0.98.61:37997)
0925 16:27:28.007220 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.007225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.007242 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:27:28.007295 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.007298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.007361 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.007478 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.008130 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.008817 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.009846 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.010173 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.010281 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.013118 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.013122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.013125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.013149 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.015935 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.016515 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.016519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.016520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.016609 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.016912 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.016915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.016916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.017170 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.017255 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.017875 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.018060 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":89}
I20260925 16:27:28.022698 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Bootstrap starting.
I20260925 16:27:28.026379 7035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:28.027596 7035 log.cc:824] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:28.030870 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: No bootstrap required, opened a new log
I20260925 16:27:28.035190 7035 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } }
I20260925 16:27:28.035562 7035 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:28.035630 7035 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3a70b12add4fa6b8e6351fff12935c, State: Initialized, Role: FOLLOWER
I20260925 16:27:28.036288 7035 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } }
I20260925 16:27:28.037101 7041 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } } }
I20260925 16:27:28.037312 7041 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:28.038120 7035 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:27:28.044775 7052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:28.044931 7052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:28.048836 6967 master_runner.cc:428] Detected that this master bb3a70b12add4fa6b8e6351fff12935c is joining an existing cluster
I20260925 16:27:28.048954 6967 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:37997
I20260925 16:27:28.057058 6694 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260925 16:27:28.062988 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.052407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:50724)
0925 16:27:28.053192 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.053213 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.053400 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.053749 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.053798 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.054257 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.054531 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.054542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.055553 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.055556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.056231 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.056240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.056920 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.056923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.057023 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.057890 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.057906 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.058469 (+ 563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.060901 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.060906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.060909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.061193 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.061528 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.061531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.061533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.061672 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.062281 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.062283 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.062388 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.062608 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.062794 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":53,"thread_start_us":163,"threads_started":1}
I20260925 16:27:28.063066 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.052543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.0.98.62:34097)
0925 16:27:28.052711 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.052716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.053353 (+ 637us) server_negotiation.cc:407] Connection header received
0925 16:27:28.053539 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.053542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.053608 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.053685 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.054699 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.055455 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.056344 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.056634 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.056738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.058665 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.058669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.058672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.058705 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.060705 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.061308 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.061312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.061314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.061412 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.061851 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.061855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.061857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.062147 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.062216 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.062715 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.062883 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:27:28.066336 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.052914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37302)
0925 16:27:28.053248 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.053264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.053436 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.053811 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.053814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.054169 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.054323 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.054330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.055376 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.055381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.056372 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.056386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.057038 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.057041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.057125 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.060337 (+ 3212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.060355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.060920 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.064177 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.064180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.064182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.064418 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.064746 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.064749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.064751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.064831 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.065545 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.065548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.065643 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.065831 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.066039 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":3,"thread_start_us":149,"threads_started":1}
I20260925 16:27:28.066356 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.052700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.0.98.61:37997)
0925 16:27:28.052819 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.052824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.053360 (+ 536us) server_negotiation.cc:407] Connection header received
0925 16:27:28.053529 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.053532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.053573 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.053621 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.054433 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.055222 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.056518 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.056816 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.056907 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.061289 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.061295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.061303 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.061335 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.064025 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.064553 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.064558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.064560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.064643 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.064963 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.064969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.064971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.065342 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.065501 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.065953 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.066136 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":51}
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 16:27:28.083995 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.071375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.0.98.61:37997)
0925 16:27:28.071534 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.071540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.072626 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 16:27:28.072885 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.072889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.072952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.073029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.074066 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.074737 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.076168 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.076420 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.076562 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.079709 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.079714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.079716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.079750 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.081297 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.082083 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.082090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.082093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.082266 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.082716 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.082720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.082721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.083005 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.083115 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.083638 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.083806 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:27:28.084012 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.071345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37310)
0925 16:27:28.072437 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.072489 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.072712 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.073102 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.073109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.073636 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.073925 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.073935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.074896 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.074900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.075917 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.075935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.076551 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.076555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.076706 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.077441 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.077454 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.079460 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.081473 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.081493 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.081502 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.081904 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.082473 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.082476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.082478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.082590 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.083143 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.083153 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.083346 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.083536 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.083828 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260925 16:27:28.086620 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:37997: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 37997
}
attrs {
promote: true
}
}
dest_uuid: "538581326ec446cfa4f4f3373754f1fb"
cas_config_opid_index: -1
I20260925 16:27:28.087649 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } }
I20260925 16:27:28.090596 7065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } }
I20260925 16:27:28.093101 7009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:28.094035 7065 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:37997
I20260925 16:27:28.094532 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } }
I20260925 16:27:28.094830 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:28.095116 7009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Refusing update from remote peer 538581326ec446cfa4f4f3373754f1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 16:27:28.096192 6967 master.cc:537] Master@127.0.98.61:37997 shutting down...
I20260925 16:27:28.096515 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:27:28.099310 6490 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 16:27:28.100708 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb -> Peer bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61:37997): Couldn't send request to peer bb3a70b12add4fa6b8e6351fff12935c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 16:27:28.102155 6967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:28.102735 6967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:28.102955 6967 tablet_replica.cc:330] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 16:27:28.112541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33391
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:27:28.119879 6967 master.cc:552] Master@127.0.98.61:37997 shutdown complete.
I20260925 16:27:28.119966 6967 master_runner.cc:305] Clearing existing system tablet
I20260925 16:27:28.120254 6967 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:28.120321 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:28.122007 6967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:28.123361 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.123700 6967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:28.123881 6967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:28.123929 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
uuid: "bb3a70b12add4fa6b8e6351fff12935c"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "6eeeae7d5106b4ff4f65820bae7813ee"
server_key_iv: "85b55c3eda2205045ac799488e1e37b9"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.127403 6967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:28.131074 6967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 16:27:28.131250 6967 log.cc:1196] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 16:27:28.131552 6967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Deleting consensus metadata
I20260925 16:27:28.131745 6967 master_runner.cc:315] Copying system tablet from 127.0.98.62:34097
I20260925 16:27:28.133461 6967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34097
I20260925 16:27:28.147152 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.134399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:50734)
0925 16:27:28.135120 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.135153 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.135329 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.135746 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.135750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.136020 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.136204 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.136213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.137723 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.137727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.139158 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.139183 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.140059 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.140064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.140269 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.141063 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.141081 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.141679 (+ 598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.144370 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.144375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.144377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.144747 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.145356 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.145362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.145365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.145653 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.146343 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.146349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.146522 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.146717 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.146936 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":273,"threads_started":1}
I20260925 16:27:28.147202 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.134548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.98.62:34097)
0925 16:27:28.134754 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.134761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.135297 (+ 536us) server_negotiation.cc:407] Connection header received
0925 16:27:28.135460 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.135465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.135547 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.135696 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.136387 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.137490 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.139450 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.139864 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.140024 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.141981 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.141986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.141989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.142029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.144156 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.144894 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.144898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.144900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.145030 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.145822 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.145826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.145828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.146147 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.146264 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.146795 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.146999 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 16:27:28.148679 6523 tablet_copy_service.cc:140] P 538581326ec446cfa4f4f3373754f1fb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bb3a70b12add4fa6b8e6351fff12935c ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50734)
I20260925 16:27:28.148947 6523 tablet_copy_service.cc:161] P 538581326ec446cfa4f4f3373754f1fb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bb3a70b12add4fa6b8e6351fff12935c at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50734: session id = bb3a70b12add4fa6b8e6351fff12935c-00000000000000000000000000000000
I20260925 16:27:28.151356 6523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 16:27:28.153965 6967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 16:27:28.157462 6967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: tablet copy: Starting download of 0 data blocks...
I20260925 16:27:28.157749 6967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: tablet copy: Starting download of 1 WAL segments...
I20260925 16:27:28.161037 6967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 16:27:28.164906 6523 tablet_copy_service.cc:342] P 538581326ec446cfa4f4f3373754f1fb: Request end of tablet copy session bb3a70b12add4fa6b8e6351fff12935c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50734
I20260925 16:27:28.165179 6523 tablet_copy_service.cc:433] P 538581326ec446cfa4f4f3373754f1fb: ending tablet copy session bb3a70b12add4fa6b8e6351fff12935c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bb3a70b12add4fa6b8e6351fff12935c
I20260925 16:27:28.176915 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:28.180011 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:28.180569 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:28.180819 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:28.183004 6967 server_base.cc:1034] running on GCE node
I20260925 16:27:28.183384 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:28.183854 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:28.185079 6967 hybrid_clock.cc:648] HybridClock initialized: now 1790353648185061 us; error 40 us; skew 500 ppm
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:28.187450 6967 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 16:27:28.188097 6967 webserver.cc:460] Webserver started at http://127.0.98.61:38547/ using document root <none> and password file <none>
I20260925 16:27:28.188336 6967 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:28.188390 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:28.190160 6967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:28.191538 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.191984 6967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:28.192098 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
uuid: "bb3a70b12add4fa6b8e6351fff12935c"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "6eeeae7d5106b4ff4f65820bae7813ee"
server_key_iv: "85b55c3eda2205045ac799488e1e37b9"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.192209 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:28.214175 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:28.216969 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:28.233906 6967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37997
I20260925 16:27:28.234998 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:27:28.237577 7141 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:27:28.237993 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37997 every 8 connection(s)
I20260925 16:27:28.239194 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Bootstrap starting.
I20260925 16:27:28.255533 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:27:28.256052 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: Bootstrap complete.
I20260925 16:27:28.256963 7141 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } }
I20260925 16:27:28.257205 7141 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb3a70b12add4fa6b8e6351fff12935c, State: Initialized, Role: LEARNER
I20260925 16:27:28.257344 7141 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } }
I20260925 16:27:28.258159 7141 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:28.258229 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } }
I20260925 16:27:28.258772 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: LEARNER
I20260925 16:27:28.262734 7155 catalog_manager.cc:1259] Loaded cluster ID: 3c397eaee7914c7eb7a551bd81a560ba
I20260925 16:27:28.262837 7155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: loading cluster ID for follower catalog manager: success
I20260925 16:27:28.264662 7155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: acquiring CA information for follower catalog manager: success
I20260925 16:27:28.266187 7155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 16:27:28.306880 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:28.307336 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:28.307389 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:28.312160 7069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:28.312278 7069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:28.312297 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:28.312314 7069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:28.312330 7069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:28.319026 7069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:28.320730 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:28.322228 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:28.334044 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:28.334221 7069 server_base.cc:1034] running on GCE node
W20260925 16:27:28.334134 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:28.334095 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:28.335270 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:28.336995 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:28.338210 7069 hybrid_clock.cc:648] HybridClock initialized: now 1790353648338189 us; error 68 us; skew 500 ppm
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:28.342067 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 16:27:28.343652 7069 webserver.cc:460] Webserver started at http://127.0.98.60:44019/ using document root <none> and password file <none>
I20260925 16:27:28.344519 7069 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:28.344645 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:28.344935 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:28.347919 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/instance:
uuid: "9b408cfaa09c43a683c8fbbabb148db0"
format_stamp: "Formatted at 2026-09-25 16:27:28 on dist-test-slave-31kt"
server_key: "a2d336a5ee1f3a96c494b93d900d935a"
server_key_iv: "3790b2c5e82b987942b4aa28ea5f80f6"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.348735 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal/instance:
uuid: "9b408cfaa09c43a683c8fbbabb148db0"
format_stamp: "Formatted at 2026-09-25 16:27:28 on dist-test-slave-31kt"
server_key: "a2d336a5ee1f3a96c494b93d900d935a"
server_key_iv: "3790b2c5e82b987942b4aa28ea5f80f6"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.354801 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 16:27:28.358929 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.361212 7069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:28.361433 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
uuid: "9b408cfaa09c43a683c8fbbabb148db0"
format_stamp: "Formatted at 2026-09-25 16:27:28 on dist-test-slave-31kt"
server_key: "a2d336a5ee1f3a96c494b93d900d935a"
server_key_iv: "3790b2c5e82b987942b4aa28ea5f80f6"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.361608 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:28.373906 6828 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260925 16:27:28.376233 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:28.380208 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:28.380432 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:28.388463 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33391
I20260925 16:27:28.388831 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33391 every 8 connection(s)
I20260925 16:27:28.390018 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:27:28.393878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 7069
I20260925 16:27:28.395318 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:28.401899 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:28.415676 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.396280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.98.60:33391)
0925 16:27:28.397256 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.397283 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.397348 (+ 65us) server_negotiation.cc:407] Connection header received
0925 16:27:28.398277 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.398302 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.398775 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.399169 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.400497 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.402029 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.403514 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.404217 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.404467 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.408231 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.408239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.408254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.408295 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.411098 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.411850 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.411857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.411864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.412075 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.412461 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.412465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.412467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.413080 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.413291 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.414010 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.414456 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":265,"threads_started":1}
I20260925 16:27:28.419234 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.405156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.0.98.62:34097)
0925 16:27:28.405347 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.405353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.405968 (+ 615us) server_negotiation.cc:407] Connection header received
0925 16:27:28.406177 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.406181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.406252 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.406342 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.407360 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.408035 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.409312 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.409981 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.410191 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.413642 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.413647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.413661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.413706 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.415370 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.417262 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.417267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.417269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.417451 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.417908 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.417913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.417914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.418212 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.418336 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.418746 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.419027 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 16:27:28.419251 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.405028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:50740)
0925 16:27:28.405734 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.405855 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.406056 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.406451 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.406459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.406996 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.407180 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.407187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.408190 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.408194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.409146 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.409157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.410198 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.410201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.410432 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.411102 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.411119 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.413370 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.415509 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.415520 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.415528 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.417029 (+ 1501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.417609 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.417613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.417616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.417747 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.418342 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.418348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.418515 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.418659 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.419019 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":256,"threads_started":1}
I20260925 16:27:28.421203 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:27:28.432981 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.422416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.98.61:37997)
0925 16:27:28.422912 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.422916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.422931 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:27:28.422972 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.422974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.423026 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.423142 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.423926 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.424788 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.426111 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.426381 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.426497 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.429639 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.429642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.429645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.429675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.431158 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.431654 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.431658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.431659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.431758 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.432011 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.432015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.432016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.432225 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.432314 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.432575 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.432784 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":180,"threads_started":1}
I20260925 16:27:28.432979 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.422286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37324)
0925 16:27:28.422450 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.422466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.422566 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.423316 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.423320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.423559 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.423718 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.423725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.424995 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.424999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.425903 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.425921 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.426628 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.426632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.426757 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.427581 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.427604 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.429432 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.431314 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.431319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.431321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.431557 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.431844 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.431847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.431849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.431926 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.432315 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.432319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.432424 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.432514 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.432817 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 16:27:28.434643 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:27:28.446239 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.435429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47138)
0925 16:27:28.435633 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.435653 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.435764 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.436202 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.436205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.436424 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.436536 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.436542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.437954 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.437958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.438583 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.438596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.439267 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.439270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.439358 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.439996 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.440009 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.441916 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.444123 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.444126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.444128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.444361 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.444765 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.444768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.444770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.444857 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.445474 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.445478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.445678 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.445901 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.446084 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 16:27:28.446452 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.435590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.0.98.60:33391)
0925 16:27:28.435752 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.435756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.435773 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:27:28.435947 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.435950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.436007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.436106 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.436673 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.437731 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.438763 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.439060 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.439174 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.442149 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.442153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.442156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.442189 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.443939 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.444480 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.444541 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.444543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.444642 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.445000 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.445003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.445004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.445295 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.445389 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.446001 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.446195 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:27:28.451062 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Bootstrap starting.
I20260925 16:27:28.455651 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:28.456933 7223 log.cc:824] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:28.460693 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: No bootstrap required, opened a new log
I20260925 16:27:28.465229 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } }
I20260925 16:27:28.465605 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:28.465662 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b408cfaa09c43a683c8fbbabb148db0, State: Initialized, Role: FOLLOWER
I20260925 16:27:28.466351 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } }
I20260925 16:27:28.467344 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } } }
I20260925 16:27:28.467561 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:28.468268 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:27:28.475720 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:28.475883 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:28.479897 7069 master_runner.cc:428] Detected that this master 9b408cfaa09c43a683c8fbbabb148db0 is joining an existing cluster
I20260925 16:27:28.480003 7069 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33391
I20260925 16:27:28.495893 7229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:28.496254 7229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } }
I20260925 16:27:28.497634 7229 leader_election.cc:288] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61:37997)
I20260925 16:27:28.498389 6511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b408cfaa09c43a683c8fbbabb148db0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538581326ec446cfa4f4f3373754f1fb" is_pre_election: true
I20260925 16:27:28.498389 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b408cfaa09c43a683c8fbbabb148db0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3a70b12add4fa6b8e6351fff12935c" is_pre_election: true
I20260925 16:27:28.498672 7110 raft_consensus.cc:1799] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Handling vote request from an unknown peer 9b408cfaa09c43a683c8fbbabb148db0
I20260925 16:27:28.498831 7110 raft_consensus.cc:2396] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9b408cfaa09c43a683c8fbbabb148db0 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 16:27:28.499542 7171 leader_election.cc:302] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b408cfaa09c43a683c8fbbabb148db0; no voters: 538581326ec446cfa4f4f3373754f1fb, bb3a70b12add4fa6b8e6351fff12935c
I20260925 16:27:28.500234 7229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:28.501644 7229 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 16:27:28.514225 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.484393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.0.98.60:33391)
0925 16:27:28.484500 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.484504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.484762 (+ 258us) server_negotiation.cc:407] Connection header received
0925 16:27:28.485085 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.485090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.485137 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.485186 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.486302 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.487142 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.488154 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.488365 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.488546 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.490410 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.490415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.490418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.490440 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.512005 (+ 21565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.512468 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.512472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.512474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.512552 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.512840 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.512843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.512845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.513310 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.513396 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.513797 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.514000 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":49}
I20260925 16:27:28.514869 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.483659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47148)
0925 16:27:28.484634 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.484650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.484936 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.485407 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.485410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.485876 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.486153 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.486163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.487289 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.487293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.488008 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.488021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.488762 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.488766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.488881 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.489595 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.489615 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.490262 (+ 647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.512128 (+ 21866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.512131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.512134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.512360 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.512652 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.512655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.512657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.512743 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.513463 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.513466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.513584 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.514531 (+ 947us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.514700 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":78,"thread_start_us":172,"threads_started":1}
I20260925 16:27:28.515623 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.484229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.0.98.62:34097)
0925 16:27:28.484372 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.484377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.484397 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:28.484457 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.484460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.484518 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.484601 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.485297 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.486174 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.489839 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.490222 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.490323 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.511720 (+ 21397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.511727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.511731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.511774 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.513646 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.514227 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.514231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.514233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.514333 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.514634 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.514637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.514639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.514876 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.514954 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.515301 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.515445 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:27:28.515707 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.483574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.98.61:37997)
0925 16:27:28.483792 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.483797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.484997 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 16:27:28.485194 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.485198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.485263 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.485320 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.486282 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.487151 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.489398 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.489826 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.489954 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.510900 (+ 20946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.510906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.510910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.510952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.513079 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.513736 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.513741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.513743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.513879 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.514199 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.514215 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.514217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.514479 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.514562 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.515370 (+ 808us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.515550 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 16:27:28.516225 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.483506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:50744)
0925 16:27:28.484183 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.484202 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.484349 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.484636 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.484639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.484948 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.485146 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.485154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.486311 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.486314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.489624 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.489634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.490501 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.490504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.490613 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.510802 (+ 20189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.510829 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.511415 (+ 586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.513813 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.513816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.513818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.514110 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.514440 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.514443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.514445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.514539 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.515024 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.515026 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.515118 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.515858 (+ 740us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.516037 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":488,"thread_start_us":175,"threads_started":1}
I20260925 16:27:28.516961 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.483754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37338)
0925 16:27:28.484892 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.484905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.485063 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.485465 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.485468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.485884 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.486116 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.486124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.487300 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.487304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.489213 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.489227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.490096 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.490100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.490217 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.490927 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.490943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.510612 (+ 19669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.513295 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.513300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.513302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.513607 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.513996 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.513999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.514001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.514097 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.515081 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.515083 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.515153 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.515906 (+ 753us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.516809 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":766,"thread_start_us":106,"threads_started":1}
I20260925 16:27:28.522506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6565
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:27:28.532146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34861
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34097
--webserver_port=39171
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:27:28.533715 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.521306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.0.98.60:33391)
0925 16:27:28.521428 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.521433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.521448 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:27:28.521690 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.521693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.521816 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.521911 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.522512 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.524100 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.525396 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.525729 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.525866 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.529142 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.529147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.529150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.529188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.531158 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.531835 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.531842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.531844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.532030 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.532446 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.532450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.532452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.532744 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.532848 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.533344 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.533508 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:27:28.533726 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.521199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47160)
0925 16:27:28.521345 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.521362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.521461 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.521948 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.521952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.522223 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.522361 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.522369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.524287 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.524292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.525195 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.525215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.525925 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.525930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.526058 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.526783 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.526800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.528887 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.531348 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.531352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.531356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.531664 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.532186 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.532190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.532192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.532306 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.532913 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.532919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.533116 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.533262 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.533504 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 16:27:28.535840 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33391: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33391
}
attrs {
promote: true
}
}
dest_uuid: "538581326ec446cfa4f4f3373754f1fb"
cas_config_opid_index: 5
I20260925 16:27:28.536690 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } }
I20260925 16:27:28.538439 7065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.540807 7065 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33391
I20260925 16:27:28.541412 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.541565 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:28.543006 7069 master.cc:537] Master@127.0.98.60:33391 shutting down...
I20260925 16:27:28.543754 7196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Refusing update from remote peer 538581326ec446cfa4f4f3373754f1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 16:27:28.544739 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:27:28.547402 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb -> Peer 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60:33391): Couldn't send request to peer 9b408cfaa09c43a683c8fbbabb148db0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 16:27:28.549649 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.539271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37350)
0925 16:27:28.539756 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.539773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.539887 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.544722 (+ 4835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.544726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.544757 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:28.544888 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.544896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.546192 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.546196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.547473 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.547487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.549048 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.549052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.549169 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.549233 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.549311 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.549430 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386}
I20260925 16:27:28.550608 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.539583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.0.98.61:37997)
0925 16:27:28.543834 (+ 4251us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.543843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.543870 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:27:28.543979 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.543983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.544094 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.544258 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:28.545164 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.546031 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.547833 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.548563 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.548671 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.548919 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.550145 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.550289 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4096}
I20260925 16:27:28.550562 7069 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:28.550987 7069 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:28.551126 7069 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: stopping tablet replica
I20260925 16:27:28.552403 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Refusing update from remote peer 538581326ec446cfa4f4f3373754f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 16:27:28.553268 7250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 16:27:28.557828 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.563110 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538581326ec446cfa4f4f3373754f1fb. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.563345 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: LEARNER
I20260925 16:27:28.563484 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: true } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.563581 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: LEARNER
I20260925 16:27:28.563872 7064 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: attempting to promote NON_VOTER bb3a70b12add4fa6b8e6351fff12935c to VOTER
I20260925 16:27:28.564627 7064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } }
I20260925 16:27:28.566537 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.566053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47170)
0925 16:27:28.566322 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.566419 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260925 16:27:28.566869 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 LEARNER]: Refusing update from remote peer 538581326ec446cfa4f4f3373754f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 16:27:28.567384 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 16:27:28.568693 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.567680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47176)
0925 16:27:28.568478 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.568571 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":438,"threads_started":1}
W20260925 16:27:28.568816 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb -> Peer 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60:33391): Couldn't send request to peer 9b408cfaa09c43a683c8fbbabb148db0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 16:27:28.571161 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.574712 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.574929 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:28.575445 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.575555 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:28.575544 7069 master.cc:552] Master@127.0.98.60:33391 shutdown complete.
I20260925 16:27:28.575618 7069 master_runner.cc:305] Clearing existing system tablet
I20260925 16:27:28.575910 7069 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:28.575989 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:28.576954 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.577431 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 16:27:28.579627 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.579831 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:28.580648 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.581081 7069 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:28.581272 7069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 16:27:28.581327 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
uuid: "9b408cfaa09c43a683c8fbbabb148db0"
format_stamp: "Formatted at 2026-09-25 16:27:28 on dist-test-slave-31kt"
server_key: "a2d336a5ee1f3a96c494b93d900d935a"
server_key_iv: "3790b2c5e82b987942b4aa28ea5f80f6"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.584389 7069 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:28.588146 7069 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 16:27:28.588310 7069 log.cc:1196] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 16:27:28.588573 7069 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Deleting consensus metadata
I20260925 16:27:28.589193 7069 master_runner.cc:315] Copying system tablet from 127.0.98.62:34097
I20260925 16:27:28.590783 7069 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34097
I20260925 16:27:28.602972 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.591610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:50756)
0925 16:27:28.592073 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.592092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.592348 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.592762 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.592765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.593119 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.593299 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.593493 (+ 194us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.595034 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.595038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.595975 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.595987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.596720 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.596724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.596831 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.597609 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.597625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.598241 (+ 616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.600848 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.600853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.600856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.601221 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.601619 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.601623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.601625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.601735 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.602246 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.602249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.602405 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.602511 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.602738 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":140,"threads_started":1}
I20260925 16:27:28.603109 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.591741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.0.98.62:34097)
0925 16:27:28.591911 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.591915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.592207 (+ 292us) server_negotiation.cc:407] Connection header received
0925 16:27:28.592473 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.592477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.592580 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.592715 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.593936 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.594822 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.596171 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.596520 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.596674 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.598462 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.598466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.598469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.598511 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.600636 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.601367 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.601371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.601374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.601497 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.601869 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.601872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.601874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.602135 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.602252 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.602695 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.602921 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:27:28.604162 6523 tablet_copy_service.cc:140] P 538581326ec446cfa4f4f3373754f1fb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9b408cfaa09c43a683c8fbbabb148db0 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50756)
I20260925 16:27:28.604312 6523 tablet_copy_service.cc:161] P 538581326ec446cfa4f4f3373754f1fb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9b408cfaa09c43a683c8fbbabb148db0 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50756: session id = 9b408cfaa09c43a683c8fbbabb148db0-00000000000000000000000000000000
I20260925 16:27:28.605722 6523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 16:27:28.607816 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 16:27:28.611920 7069 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: tablet copy: Starting download of 0 data blocks...
I20260925 16:27:28.612500 7069 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: tablet copy: Starting download of 1 WAL segments...
I20260925 16:27:28.615332 7069 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 16:27:28.618521 6523 tablet_copy_service.cc:342] P 538581326ec446cfa4f4f3373754f1fb: Request end of tablet copy session 9b408cfaa09c43a683c8fbbabb148db0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50756
I20260925 16:27:28.618670 6523 tablet_copy_service.cc:433] P 538581326ec446cfa4f4f3373754f1fb: ending tablet copy session 9b408cfaa09c43a683c8fbbabb148db0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9b408cfaa09c43a683c8fbbabb148db0
I20260925 16:27:28.633692 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:28.636302 7265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:28.636620 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:28.636906 7069 server_base.cc:1034] running on GCE node
W20260925 16:27:28.636332 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:28.638161 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:28.638566 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:28.639745 7069 hybrid_clock.cc:648] HybridClock initialized: now 1790353648639735 us; error 39 us; skew 500 ppm
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:28.641681 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 16:27:28.642269 7069 webserver.cc:460] Webserver started at http://127.0.98.60:41797/ using document root <none> and password file <none>
I20260925 16:27:28.642472 7069 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:28.642519 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:28.644075 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:28.645301 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.645690 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:28.645861 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
uuid: "9b408cfaa09c43a683c8fbbabb148db0"
format_stamp: "Formatted at 2026-09-25 16:27:28 on dist-test-slave-31kt"
server_key: "a2d336a5ee1f3a96c494b93d900d935a"
server_key_iv: "3790b2c5e82b987942b4aa28ea5f80f6"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.645963 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:28.674167 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:28.676358 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:28.681425 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33391
I20260925 16:27:28.682317 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:27:28.682725 7324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33391 every 8 connection(s)
I20260925 16:27:28.684701 6962 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260925 16:27:28.684926 7325 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:27:28.686501 7325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Bootstrap starting.
I20260925 16:27:28.702705 7325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:27:28.703167 7325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Bootstrap complete.
I20260925 16:27:28.704044 7325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } }
I20260925 16:27:28.704246 7325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b408cfaa09c43a683c8fbbabb148db0, State: Initialized, Role: LEARNER
I20260925 16:27:28.704370 7325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } }
I20260925 16:27:28.705026 7325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:28.704914 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:28.705265 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:27:28.709137 7339 catalog_manager.cc:1259] Loaded cluster ID: 3c397eaee7914c7eb7a551bd81a560ba
I20260925 16:27:28.709292 7339 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: loading cluster ID for follower catalog manager: success
I20260925 16:27:28.711223 7339 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: acquiring CA information for follower catalog manager: success
I20260925 16:27:28.712666 7339 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 16:27:28.727423 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:28.727880 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:28.727952 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:28.732929 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:28.733042 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:28.733135 7249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:28.739830 7249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39171
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:28.741637 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:28.743163 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:28.757833 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:28.757833 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:28.757838 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:28.758586 7249 server_base.cc:1034] running on GCE node
I20260925 16:27:28.759225 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:28.760182 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:28.761658 7249 hybrid_clock.cc:648] HybridClock initialized: now 1790353648761609 us; error 65 us; skew 500 ppm
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:28.766179 7249 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:27:28.767479 7249 webserver.cc:460] Webserver started at http://127.0.98.1:39171/ using document root <none> and password file <none>
I20260925 16:27:28.768203 7249 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:28.768302 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:28.773598 7249 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:27:28.776712 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.778942 7249 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:28.779155 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "30cce7aad35d402096abc60e5d7a4563"
format_stamp: "Formatted at 2026-09-25 16:27:26 on dist-test-slave-31kt"
server_key: "0e7b5e53513c8105589d0c3eeb1e572a"
server_key_iv: "852ac1242ce8c5509242b4c447d331a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:28.779860 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:28.803692 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:28.808501 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:28.808779 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:28.809648 7249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:28.811316 7249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:28.811399 7249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.811476 7249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:28.811513 7249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:28.824410 7249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34861
I20260925 16:27:28.824560 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34861 every 8 connection(s)
I20260925 16:27:28.826493 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:28.834998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 7249
I20260925 16:27:28.835291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6699
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:27:28 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:27:28.845832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39793
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34097
--webserver_port=39979
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:27:28.849411 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.829855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38561 (local address 127.0.98.62:34097)
0925 16:27:28.830005 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.830010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.831322 (+ 1312us) server_negotiation.cc:407] Connection header received
0925 16:27:28.832985 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.832990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.833073 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.833190 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.836324 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.837135 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.839556 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.839975 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.840078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.845079 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.845082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.845085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.845116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.847542 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.848050 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.848054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.848055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.848142 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.848450 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.848454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.848455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.848685 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.848770 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.849086 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.849233 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:27:28.850687 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.830277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.98.1:39955)
0925 16:27:28.831798 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.831813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.832583 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.833584 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.833598 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.834521 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.835863 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.835883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.840109 (+ 4226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.840113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.840907 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.840919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.841561 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.841564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.841689 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.842241 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.842255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.844475 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.846781 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.846790 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.846816 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.847249 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.847588 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.847591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.847593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.847807 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.848344 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.848352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.848759 (+ 407us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.848979 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.849395 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"mutex_wait_us":259,"spinlock_wait_cycles":83328,"thread_start_us":197,"threads_started":1}
I20260925 16:27:28.850719 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.829548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56821 (local address 127.0.98.60:33391)
0925 16:27:28.829911 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.829916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.832272 (+ 2356us) server_negotiation.cc:407] Connection header received
0925 16:27:28.833000 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.833003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.833073 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.833157 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.836096 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.836935 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.839538 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.839935 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.840030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.845402 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.845408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.845412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.845916 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.848067 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.848755 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.848759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.848761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.848861 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.849157 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.849160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.849161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.849459 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.849545 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.850324 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.850506 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260925 16:27:28.850986 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.830283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:38561)
0925 16:27:28.831060 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.831141 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.832583 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.833582 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.833598 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.834490 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.836154 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.836160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.837298 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.837301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.839308 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.839331 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.840323 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.840327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.840584 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.841524 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.841542 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.844393 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.847660 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.847663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.847664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.847942 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.848260 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.848263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.848265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.848350 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.848830 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.848833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.848915 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.849612 (+ 697us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.849802 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":269,"spinlock_wait_cycles":24704,"thread_start_us":234,"threads_started":1}
I20260925 16:27:28.851805 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.829548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39955 (local address 127.0.98.61:37997)
0925 16:27:28.829729 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:28.829735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:28.832067 (+ 2332us) server_negotiation.cc:407] Connection header received
0925 16:27:28.833332 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:28.833335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:28.833389 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:28.833476 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:28.839160 (+ 5684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.839923 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.841067 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.841322 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.841419 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.844638 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:28.844643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:28.844646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:28.844679 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:28.846595 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.847386 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.847390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.847392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.847489 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:28.847921 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:28.847925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:28.847926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:28.848197 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:28.848287 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:28.851488 (+ 3201us) server_negotiation.cc:299] Negotiation successful
0925 16:27:28.851643 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 16:27:28.852289 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:28.830277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.98.1:56821)
0925 16:27:28.832108 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:28.832121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:28.832656 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:28.833581 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:28.833598 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:28.834493 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:28.835781 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.835814 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.837145 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.837149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.839302 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:28.839323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:28.840290 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:28.840294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:28.840574 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:28.841433 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:28.841459 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:28.844850 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:28.848286 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.848291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.848294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.848613 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.848969 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:28.848972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:28.848974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:28.849063 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:28.849965 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:28.849968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:28.850115 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 16:27:28.851559 (+ 1444us) client_negotiation.cc:240] Negotiation successful
0925 16:27:28.851800 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"mutex_wait_us":520,"spinlock_wait_cycles":4352,"thread_start_us":753,"threads_started":1}
I20260925 16:27:28.852571 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260925 16:27:28.852464 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37997
I20260925 16:27:28.852849 7469 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:28.852849 7470 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:28.853628 7470 heartbeater.cc:511] Master 127.0.98.61:37997 requested a full tablet report, sending...
I20260925 16:27:28.854152 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33391
I20260925 16:27:28.854312 7468 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:28.854637 7469 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260925 16:27:28.854640 7468 heartbeater.cc:511] Master 127.0.98.60:33391 requested a full tablet report, sending...
I20260925 16:27:28.856058 6503 ts_manager.cc:194] Re-registered known tserver with Master: 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861)
I20260925 16:27:28.856150 7105 ts_manager.cc:194] Registered new tserver with Master: 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861)
I20260925 16:27:28.856150 7289 ts_manager.cc:194] Registered new tserver with Master: 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861)
I20260925 16:27:28.856901 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38561
W20260925 16:27:28.995698 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:28.996125 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:28.996181 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:29.001461 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:29.001622 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:29.001699 7476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:29.009087 7476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39979
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:29.011140 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:29.012856 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:29.027899 7481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:29.027899 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:29.027902 7482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:29.028033 7476 server_base.cc:1034] running on GCE node
I20260925 16:27:29.028862 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:29.029726 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:29.031132 7476 hybrid_clock.cc:648] HybridClock initialized: now 1790353649031097 us; error 75 us; skew 500 ppm
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:29.034549 7476 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:27:29.035775 7476 webserver.cc:460] Webserver started at http://127.0.98.2:39979/ using document root <none> and password file <none>
I20260925 16:27:29.036535 7476 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:29.036664 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:29.043787 7476 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:29.047416 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:29.050002 7476 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:27:29.050242 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "d10d58d5c2f2421bae4e7d593b9776fa"
server_key_iv: "930dc5156c2c607a99ac97e80f291a18"
server_key_version: "encryptionkey@0"
I20260925 16:27:29.051110 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:29.067505 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.062624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47178)
0925 16:27:29.062823 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.062845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.063026 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.063441 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.063443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.063458 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:29.063603 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.063612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.064915 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.064919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.065804 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.065836 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.066893 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.066897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.067021 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.067053 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.067132 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.067284 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 16:27:29.067718 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.062744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.0.98.60:33391)
0925 16:27:29.062939 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.062943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.062964 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:27:29.063226 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.063229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.063312 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.063422 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:29.063775 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.064696 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.065963 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.066703 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.066849 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.067419 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.067466 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.067588 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 16:27:29.068884 7299 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 16:27:29.069581 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538581326ec446cfa4f4f3373754f1fb. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:29.069746 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: This master's current role is: LEARNER
I20260925 16:27:29.071650 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:29.077000 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:29.077247 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:29.078050 7476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:29.079205 7476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:29.079276 7476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:29.079325 7476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:29.079360 7476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:29.091266 7476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39793
I20260925 16:27:29.091324 7604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39793 every 8 connection(s)
I20260925 16:27:29.092612 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:29.097602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 7476
I20260925 16:27:29.097817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6833
I20260925 16:27:29.108686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40925
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34097
--webserver_port=33437
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 16:27:29.116057 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.095585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39975 (local address 127.0.98.61:37997)
0925 16:27:29.095809 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.095840 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.099468 (+ 3628us) server_negotiation.cc:407] Connection header received
0925 16:27:29.099671 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.099674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.099738 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.099794 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.102053 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.102946 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.104869 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.105231 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.105338 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.110907 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.110911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.110913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.110943 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.112813 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.113744 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.113748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.113750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.113875 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.114356 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.114360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.114362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.114622 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.114704 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.115666 (+ 962us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.115851 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:27:29.116057 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.095386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60997 (local address 127.0.98.60:33391)
0925 16:27:29.095594 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.095600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.096975 (+ 1375us) server_negotiation.cc:407] Connection header received
0925 16:27:29.099080 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.099084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.099171 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.099269 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.101961 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.102603 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.104831 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.105214 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.105306 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.111005 (+ 5699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.111009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.111013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.111049 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.113251 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.113870 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.113879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.113881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.113985 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.114326 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.114330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.114331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.114614 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.114704 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.115666 (+ 962us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.115851 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 16:27:29.116071 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.095999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58427 (local address 127.0.98.62:34097)
0925 16:27:29.096261 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.096267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.096981 (+ 714us) server_negotiation.cc:407] Connection header received
0925 16:27:29.099505 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.099510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.099592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.099675 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.102871 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.103628 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.104918 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.105267 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.105337 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.110797 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.110804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.110807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.110859 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.113227 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.114234 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.114238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.114239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.114337 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.114621 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.114624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.114626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.114879 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.114967 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.115678 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.115849 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 16:27:29.118870 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.097902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.98.2:39975)
0925 16:27:29.099350 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.099367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.099554 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.099941 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.099944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.100464 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:29.101675 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.101683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.103214 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.103218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.104663 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.104683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.105620 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.105624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.106034 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.107092 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:29.107122 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:29.110684 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:29.113022 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.113033 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.113065 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.113612 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.113989 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.113992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.113995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.114229 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.114778 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:29.114787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:29.115308 (+ 521us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.116416 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.116925 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"mutex_wait_us":528,"spinlock_wait_cycles":98944,"thread_start_us":146,"threads_started":1}
I20260925 16:27:29.118870 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.095988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:58427)
0925 16:27:29.096740 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.096801 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.098737 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.099849 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.099852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.100464 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:29.101643 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.101660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.103807 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.103810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.104790 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.104799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.105615 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.105619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.106061 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.107096 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:29.107137 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:29.110441 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:29.113371 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.113375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.113378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.114114 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.114435 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.114438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.114440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.114528 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.115027 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:29.115030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:29.115311 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.116373 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.117104 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":508,"spinlock_wait_cycles":155904,"thread_start_us":239,"threads_started":1}
I20260925 16:27:29.119678 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.095988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.98.2:60997)
0925 16:27:29.096723 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.096806 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.098728 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.099567 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.099580 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.100301 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:29.101598 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.101636 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.102812 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.102816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.104649 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.104662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.105615 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.105619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.106077 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.107200 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:29.107212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:29.110810 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:29.113423 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.113426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.113429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.113728 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.114092 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.114095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.114097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.114216 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.114778 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:29.114787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:29.115308 (+ 521us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.116819 (+ 1511us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.116981 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":134,"spinlock_wait_cycles":32384,"thread_start_us":215,"threads_started":1}
I20260925 16:27:29.120734 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37997
I20260925 16:27:29.120743 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260925 16:27:29.121150 7607 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:29.121155 7606 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:29.120734 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33391
I20260925 16:27:29.121860 7605 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:29.122056 7606 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260925 16:27:29.122185 7607 heartbeater.cc:511] Master 127.0.98.61:37997 requested a full tablet report, sending...
I20260925 16:27:29.122087 7605 heartbeater.cc:511] Master 127.0.98.60:33391 requested a full tablet report, sending...
I20260925 16:27:29.123919 7105 ts_manager.cc:194] Registered new tserver with Master: fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793)
I20260925 16:27:29.123919 7289 ts_manager.cc:194] Registered new tserver with Master: fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793)
I20260925 16:27:29.123952 6503 ts_manager.cc:194] Re-registered known tserver with Master: fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793)
I20260925 16:27:29.125015 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58427
W20260925 16:27:29.272126 7612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:29.272473 7612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:29.272523 7612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:29.277264 7612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:29.277372 7612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:29.277439 7612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:29.284790 7612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33437
--tserver_master_addrs=127.0.98.62:34097,127.0.98.61:37997,127.0.98.60:33391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:29.286177 7612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:29.287469 7612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:29.300433 7617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:29.300433 7618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:29.300457 7620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:29.300715 7612 server_base.cc:1034] running on GCE node
I20260925 16:27:29.301419 7612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:29.302435 7612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:29.303664 7612 hybrid_clock.cc:648] HybridClock initialized: now 1790353649303633 us; error 52 us; skew 500 ppm
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:29.308012 7612 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:27:29.309419 7612 webserver.cc:460] Webserver started at http://127.0.98.3:33437/ using document root <none> and password file <none>
I20260925 16:27:29.310294 7612 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:29.310410 7612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:29.315804 7612 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:29.319067 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:29.320943 7612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:29.321147 7612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "a1991305fb884eaf93a8ec27104cada9"
format_stamp: "Formatted at 2026-09-25 16:27:27 on dist-test-slave-31kt"
server_key: "f653379e5e00bac506a099af3a40cde8"
server_key_iv: "f2881e7cd1ceabe84c3d9e88c73c589c"
server_key_version: "encryptionkey@0"
I20260925 16:27:29.321998 7612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:29.340740 7612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:29.344761 7612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:29.345002 7612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:29.345841 7612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:29.347033 7612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:29.347084 7612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:29.347146 7612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:29.347175 7612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:29.360226 7612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40925
I20260925 16:27:29.360415 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40925 every 8 connection(s)
I20260925 16:27:29.361867 7612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:29.370764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 7612
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 16:27:29 dist-test-slave-31kt krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 16:27:29.384286 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.364445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49415 (local address 127.0.98.61:37997)
0925 16:27:29.364734 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.364740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.366230 (+ 1490us) server_negotiation.cc:407] Connection header received
0925 16:27:29.367505 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.367509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.367573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.367631 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.369993 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.370810 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.372783 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.373146 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.373258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.377246 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.377251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.377255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.377292 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.380537 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.381509 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.381514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.381516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.381645 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.382429 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.382434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.382436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.382831 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.382964 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.383861 (+ 897us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.384063 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260925 16:27:29.385442 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.364582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55651 (local address 127.0.98.62:34097)
0925 16:27:29.364836 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.364840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.366485 (+ 1645us) server_negotiation.cc:407] Connection header received
0925 16:27:29.367485 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.367489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.367572 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.367630 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.369951 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.370837 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.372743 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.373024 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.373148 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.377654 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.377659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.377662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.377698 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.380715 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.382176 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.382181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.382183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.382306 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.384083 (+ 1777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.384087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.384089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.384356 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.384443 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.385097 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.385262 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260925 16:27:29.386000 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.373805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.0.98.60:33391)
0925 16:27:29.374559 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.374564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.374872 (+ 308us) server_negotiation.cc:407] Connection header received
0925 16:27:29.374923 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.374926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.374981 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.375059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.376598 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.377394 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.378655 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.378981 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.379050 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.380897 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.380901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.380904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.380932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.383397 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.384014 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.384017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.384019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.384125 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.384430 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.384433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.384495 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.384730 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.384857 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.385666 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.385828 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":80,"threads_started":1}
I20260925 16:27:29.386819 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.365185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:55651)
0925 16:27:29.366388 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.366401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.367169 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.368189 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.368198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.368799 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:29.369657 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.369682 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.370977 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.370982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.372586 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.372600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.373242 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.373246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.373523 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.374820 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:29.374834 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:29.377446 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:29.380841 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.380845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.380847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.381598 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.383036 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.383041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.383043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.383962 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.384766 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:29.384769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:29.384888 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.385652 (+ 764us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.385853 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":612,"spinlock_wait_cycles":64512,"thread_start_us":165,"threads_started":1}
I20260925 16:27:29.386830 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.365092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.98.3:49415)
0925 16:27:29.366085 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.366103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.367215 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.368189 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.368199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.368817 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:29.369662 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.369679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.371107 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.371110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.372589 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.372603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.373412 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.373416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.373529 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.374556 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:29.374585 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:29.376994 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:29.380724 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.380734 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.380760 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.381372 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.381745 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.381748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.381778 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.382033 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.382992 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:29.383005 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:29.383527 (+ 522us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.383758 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.384575 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":350,"spinlock_wait_cycles":19584,"thread_start_us":190,"threads_started":1}
I20260925 16:27:29.388296 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260925 16:27:29.388659 7742 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:29.388855 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.365091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.98.3:49923)
0925 16:27:29.365819 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.365882 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.367158 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.368189 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.368199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.368817 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:29.369664 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.369679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.370817 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.370820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.378025 (+ 7205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.378038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.381953 (+ 3915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.381958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.382082 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.382545 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:29.382558 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:29.384961 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:29.387118 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.387121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.387123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.387336 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.387656 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:29.387659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:29.387661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:29.387767 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:29.388265 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:29.388268 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:29.388404 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.388505 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.388685 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":192,"spinlock_wait_cycles":18432,"thread_start_us":151,"threads_started":1}
I20260925 16:27:29.388967 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.364685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49923 (local address 127.0.98.60:33391)
0925 16:27:29.364847 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.364852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.366028 (+ 1176us) server_negotiation.cc:407] Connection header received
0925 16:27:29.367496 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.367500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.367573 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.367631 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.369940 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.370617 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.381283 (+ 10666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.381610 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.381701 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.385163 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.385168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.385171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.385204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.386966 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.387456 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.387459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.387461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.387557 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.387863 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.387866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.387868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.388125 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.388205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.388647 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.388808 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 16:27:29.389406 7742 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260925 16:27:29.389623 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33391
I20260925 16:27:29.389806 7741 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:29.390075 7741 heartbeater.cc:511] Master 127.0.98.60:33391 requested a full tablet report, sending...
I20260925 16:27:29.390203 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37997
I20260925 16:27:29.390297 7743 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:29.390496 7743 heartbeater.cc:511] Master 127.0.98.61:37997 requested a full tablet report, sending...
I20260925 16:27:29.391122 7289 ts_manager.cc:194] Registered new tserver with Master: a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925)
I20260925 16:27:29.391575 7105 ts_manager.cc:194] Registered new tserver with Master: a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925)
I20260925 16:27:29.391711 6503 ts_manager.cc:194] Re-registered known tserver with Master: a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925)
I20260925 16:27:29.392611 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55651
I20260925 16:27:29.394301 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.373739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.0.98.62:34097)
0925 16:27:29.375086 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.375090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.375103 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:27:29.375140 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.375142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.375187 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.375270 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.377160 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.377901 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.380003 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.380374 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.380485 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.389203 (+ 8718us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x4b7e6b8. stack: 00007f8f7e4aec78 00007f8f7e4af1a9 00007f8f7e0711d7 00000000005984a6 00000000005984f9 000000000059c488 00007f8f7e4ab2ab 00007f8f7e4aa9da 00007f8f82b830c4 00007f8f82b8bfc5 00007f8f82b8e29d 00007f8f82b8a422 00007f8f82b43219 00007f8f82b43615 00007f8f82b590d3 00007f8f82b5c54b
0925 16:27:29.389308 (+ 105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.389312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.389315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.389332 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.391765 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.392370 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.392374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.392376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.392492 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.392806 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.392810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.392812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.393065 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.393198 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.393936 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.394114 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"spinlock_wait_cycles":19025152,"thread_start_us":109,"threads_started":1}
I20260925 16:27:29.395836 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.374242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.98.61:37997)
0925 16:27:29.375469 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.375473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.375486 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:27:29.375525 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.375528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.375575 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.378971 (+ 3396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.379114 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.379842 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.387740 (+ 7898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.388686 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.390266 (+ 1580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.391433 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.391437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.391440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.391465 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.393300 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.393932 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.393936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.393938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.394166 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.394530 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.394533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.394534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.394759 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.395505 (+ 746us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.395587 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.395705 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":87,"threads_started":1}
I20260925 16:27:29.398048 6503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:29.402217 6503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:29.405998 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: true } } }
I20260925 16:27:29.406204 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:29.410569 7760 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: attempting to promote NON_VOTER 9b408cfaa09c43a683c8fbbabb148db0 to VOTER
I20260925 16:27:29.410989 7760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:29.412279 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Refusing update from remote peer 538581326ec446cfa4f4f3373754f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 16:27:29.412346 7299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 LEARNER]: Refusing update from remote peer 538581326ec446cfa4f4f3373754f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 16:27:29.412976 7761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 16:27:29.413975 7761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 16:27:29.415869 7759 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.416249 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.416328 7299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.417574 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.417696 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:29.417878 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.417964 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:29.418092 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.418154 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:29.418710 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.418856 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:29.418957 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "538581326ec446cfa4f4f3373754f1fb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:29.419097 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:29.430467 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.424835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39793 (local address 127.0.0.1:42426)
0925 16:27:29.425015 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.425031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.425132 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.426190 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.426193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.426214 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:29.426333 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.426339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.427876 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.427880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.428671 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.428681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.429947 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.429951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.430094 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.430125 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.430201 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.430307 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 16:27:29.430594 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.425074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.98.2:39793)
0925 16:27:29.425547 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.425557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.425608 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:27:29.425699 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.425705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.426015 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.426490 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:29.426660 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.427769 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.428804 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.429666 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.429837 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.430020 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.430285 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.430402 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":210,"threads_started":1}
I20260925 16:27:29.434193 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.425686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40925 (local address 127.0.0.1:38430)
0925 16:27:29.426500 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.426519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.426645 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.427654 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.427658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.427676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:29.427764 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.427770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.431266 (+ 3496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.431270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.432378 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.432385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.433676 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.433680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.433822 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.433844 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.433916 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.433990 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":2,"thread_start_us":162,"threads_started":1}
I20260925 16:27:29.434079 7539 tablet_service.cc:1467] Processing CreateTablet for tablet 71f641f157f64dd9aeb99478a8963406 (DEFAULT_TABLE table=test-table [id=1661002e43be42f3b6de7d4294e40b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:29.434805 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.425446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34861 (local address 127.0.0.1:34182)
0925 16:27:29.426553 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.426570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.426667 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.427387 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.427390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.427411 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:29.427729 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.427741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.430797 (+ 3056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.430801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.432065 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.432077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.433549 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.433553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.433662 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.433727 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.433843 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.433956 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":58,"thread_start_us":160,"threads_started":1}
I20260925 16:27:29.434836 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.426174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.0.98.3:40925)
0925 16:27:29.426886 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.426898 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.426924 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:27:29.427010 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.427016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.427307 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.427496 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:29.430086 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.431126 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.432513 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.433271 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.433396 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.433619 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.434484 (+ 865us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.434654 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":125,"threads_started":1}
I20260925 16:27:29.435402 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f641f157f64dd9aeb99478a8963406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:29.435624 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.426131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.0.98.1:34861)
0925 16:27:29.426893 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.426900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.426924 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:27:29.427002 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.427007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.427228 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.427384 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:29.428826 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.429914 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.432233 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.433063 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.433192 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.433358 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.435310 (+ 1952us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.435470 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":111,"threads_started":1}
I20260925 16:27:29.438973 7675 tablet_service.cc:1467] Processing CreateTablet for tablet 71f641f157f64dd9aeb99478a8963406 (DEFAULT_TABLE table=test-table [id=1661002e43be42f3b6de7d4294e40b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:29.438977 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 71f641f157f64dd9aeb99478a8963406 (DEFAULT_TABLE table=test-table [id=1661002e43be42f3b6de7d4294e40b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:29.440347 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f641f157f64dd9aeb99478a8963406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:29.440511 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f641f157f64dd9aeb99478a8963406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:29.447721 7776 tablet_bootstrap.cc:492] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Bootstrap starting.
I20260925 16:27:29.450090 7777 tablet_bootstrap.cc:492] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Bootstrap starting.
I20260925 16:27:29.451359 7776 tablet_bootstrap.cc:654] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:29.451996 7778 tablet_bootstrap.cc:492] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Bootstrap starting.
I20260925 16:27:29.452602 7776 log.cc:824] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:29.452806 7777 tablet_bootstrap.cc:654] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:29.453815 7777 log.cc:824] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:29.455060 7778 tablet_bootstrap.cc:654] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:29.456092 7778 log.cc:824] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:29.456440 7776 tablet_bootstrap.cc:492] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: No bootstrap required, opened a new log
I20260925 16:27:29.456440 7777 tablet_bootstrap.cc:492] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: No bootstrap required, opened a new log
I20260925 16:27:29.456708 7777 ts_tablet_manager.cc:1397] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:27:29.456708 7776 ts_tablet_manager.cc:1397] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 16:27:29.459224 7778 tablet_bootstrap.cc:492] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: No bootstrap required, opened a new log
I20260925 16:27:29.459568 7778 ts_tablet_manager.cc:1397] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 16:27:29.462263 7776 raft_consensus.cc:348] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.462263 7777 raft_consensus.cc:348] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.462756 7776 raft_consensus.cc:374] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:29.462756 7777 raft_consensus.cc:374] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:29.462819 7777 raft_consensus.cc:729] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1991305fb884eaf93a8ec27104cada9, State: Initialized, Role: FOLLOWER
I20260925 16:27:29.462819 7776 raft_consensus.cc:729] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3cd4ee37e74cb2950ec70812ae8a6a, State: Initialized, Role: FOLLOWER
I20260925 16:27:29.463584 7777 consensus_queue.cc:260] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.463584 7776 consensus_queue.cc:260] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.465327 7777 ts_tablet_manager.cc:1428] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260925 16:27:29.465327 7776 ts_tablet_manager.cc:1428] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260925 16:27:29.465577 7742 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260925 16:27:29.467677 7606 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260925 16:27:29.468781 7778 raft_consensus.cc:348] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.469749 7778 raft_consensus.cc:374] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:29.469885 7778 raft_consensus.cc:729] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cce7aad35d402096abc60e5d7a4563, State: Initialized, Role: FOLLOWER
I20260925 16:27:29.471393 7778 consensus_queue.cc:260] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.472990 7778 ts_tablet_manager.cc:1428] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260925 16:27:29.473223 7469 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
W20260925 16:27:29.579926 7471 tablet.cc:2369] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:29.592411 7784 raft_consensus.cc:482] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:29.592698 7784 raft_consensus.cc:504] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.594177 7784 leader_election.cc:288] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793), a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925)
W20260925 16:27:29.595677 7608 tablet.cc:2369] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:29.599577 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.594619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39793 (local address 127.0.98.1:56665)
0925 16:27:29.595189 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.595205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.595369 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.595841 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.595844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.595865 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:29.596063 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.596070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.596885 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.596888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.598026 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.598037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.599032 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.599036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.599139 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.599165 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.599226 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.599321 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":42,"spinlock_wait_cycles":4480,"thread_start_us":269,"threads_started":1}
I20260925 16:27:29.599663 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.594724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42251 (local address 127.0.98.3:40925)
0925 16:27:29.594952 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.594958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.595293 (+ 335us) server_negotiation.cc:407] Connection header received
0925 16:27:29.595483 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.595487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.595553 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.595631 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:29.596356 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.597088 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.598369 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.598825 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.598933 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.599032 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.599429 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.599534 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 16:27:29.599643 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.594812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56665 (local address 127.0.98.2:39793)
0925 16:27:29.594949 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.594954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.595323 (+ 369us) server_negotiation.cc:407] Connection header received
0925 16:27:29.595555 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.595559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.595615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.595730 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:29.596207 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.596758 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.598197 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.598844 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.599119 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.599201 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.599284 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.599379 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:27:29.599799 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.594619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40925 (local address 127.0.98.1:42251)
0925 16:27:29.595155 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:29.595171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:29.595338 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:29.596016 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:29.596019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:29.596043 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:29.596192 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.596200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.597254 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.597257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.598213 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:29.598226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.599118 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.599128 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:29.599254 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.599298 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 16:27:29.599384 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 16:27:29.599579 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":214,"threads_started":1}
I20260925 16:27:29.600759 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f641f157f64dd9aeb99478a8963406" candidate_uuid: "30cce7aad35d402096abc60e5d7a4563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1991305fb884eaf93a8ec27104cada9" is_pre_election: true
I20260925 16:27:29.600764 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f641f157f64dd9aeb99478a8963406" candidate_uuid: "30cce7aad35d402096abc60e5d7a4563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" is_pre_election: true
I20260925 16:27:29.601238 7559 raft_consensus.cc:2454] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30cce7aad35d402096abc60e5d7a4563 in term 0.
I20260925 16:27:29.601238 7694 raft_consensus.cc:2454] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30cce7aad35d402096abc60e5d7a4563 in term 0.
I20260925 16:27:29.602008 7358 leader_election.cc:302] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cce7aad35d402096abc60e5d7a4563, fe3cd4ee37e74cb2950ec70812ae8a6a; no voters:
I20260925 16:27:29.602389 7784 raft_consensus.cc:2781] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:29.602531 7784 raft_consensus.cc:482] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:29.602574 7784 raft_consensus.cc:3037] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:29.603802 7784 raft_consensus.cc:504] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.604226 7784 leader_election.cc:288] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [CANDIDATE]: Term 1 election: Requested vote from peers fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793), a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925)
I20260925 16:27:29.605075 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f641f157f64dd9aeb99478a8963406" candidate_uuid: "30cce7aad35d402096abc60e5d7a4563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1991305fb884eaf93a8ec27104cada9"
I20260925 16:27:29.605074 7559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f641f157f64dd9aeb99478a8963406" candidate_uuid: "30cce7aad35d402096abc60e5d7a4563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a"
I20260925 16:27:29.605310 7559 raft_consensus.cc:3037] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:29.605310 7694 raft_consensus.cc:3037] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:29.607470 7559 raft_consensus.cc:2454] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30cce7aad35d402096abc60e5d7a4563 in term 1.
I20260925 16:27:29.607467 7694 raft_consensus.cc:2454] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30cce7aad35d402096abc60e5d7a4563 in term 1.
I20260925 16:27:29.608111 7356 leader_election.cc:302] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cce7aad35d402096abc60e5d7a4563, a1991305fb884eaf93a8ec27104cada9; no voters:
I20260925 16:27:29.608419 7784 raft_consensus.cc:2781] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:29.608675 7784 raft_consensus.cc:686] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 1 LEADER]: Becoming Leader. State: Replica: 30cce7aad35d402096abc60e5d7a4563, State: Running, Role: LEADER
I20260925 16:27:29.609114 7784 consensus_queue.cc:237] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } }
I20260925 16:27:29.613466 6501 catalog_manager.cc:5495] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "30cce7aad35d402096abc60e5d7a4563" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } health_report { overall_health: HEALTHY } } }
W20260925 16:27:29.615720 7744 tablet.cc:2369] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:29.631608 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.627387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.1:34861)
0925 16:27:29.627546 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.627551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.627569 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:27:29.627697 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.627700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.627777 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.627893 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:29.628443 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.629020 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.630007 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.630229 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.630324 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.630671 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:29.631171 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:29.631285 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.631365 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.631437 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:27:29.640740 7559 raft_consensus.cc:1261] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Refusing update from remote peer 30cce7aad35d402096abc60e5d7a4563: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:29.640758 7694 raft_consensus.cc:1261] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Refusing update from remote peer 30cce7aad35d402096abc60e5d7a4563: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:29.641789 7787 consensus_queue.cc:1035] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:29.642035 7784 consensus_queue.cc:1035] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:29.662212 7791 mvcc.cc:205] Tried to move back new op lower bound from 7333288548916793344 to 7333288548802265088. Current Snapshot: MvccSnapshot[applied={T|T < 7333288548916793344 or (T in {7333288548916793344})}]
I20260925 16:27:29.670552 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:29.670874 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:29.677536 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 538581326ec446cfa4f4f3373754f1fb: Sending DeleteTablet for 3 replicas of tablet 71f641f157f64dd9aeb99478a8963406
I20260925 16:27:29.678630 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f641f157f64dd9aeb99478a8963406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38430
I20260925 16:27:29.679029 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f641f157f64dd9aeb99478a8963406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42426
I20260925 16:27:29.679366 7799 tablet_replica.cc:330] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: stopping tablet replica
I20260925 16:27:29.679647 7800 tablet_replica.cc:330] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: stopping tablet replica
I20260925 16:27:29.679607 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f641f157f64dd9aeb99478a8963406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34182
I20260925 16:27:29.679883 7799 raft_consensus.cc:2229] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:29.679941 7800 raft_consensus.cc:2229] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:29.680336 7800 raft_consensus.cc:2258] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:29.680346 7799 raft_consensus.cc:2258] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:29.682507 7800 ts_tablet_manager.cc:1905] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:29.682507 7799 ts_tablet_manager.cc:1905] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:29.686172 7799 ts_tablet_manager.cc:1918] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:29.686172 7800 ts_tablet_manager.cc:1918] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:29.686326 7800 log.cc:1196] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/71f641f157f64dd9aeb99478a8963406
I20260925 16:27:29.686334 7799 log.cc:1196] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/71f641f157f64dd9aeb99478a8963406
I20260925 16:27:29.686688 7799 ts_tablet_manager.cc:1939] T 71f641f157f64dd9aeb99478a8963406 P a1991305fb884eaf93a8ec27104cada9: Deleting consensus metadata
I20260925 16:27:29.686688 7800 ts_tablet_manager.cc:1939] T 71f641f157f64dd9aeb99478a8963406 P fe3cd4ee37e74cb2950ec70812ae8a6a: Deleting consensus metadata
I20260925 16:27:29.687477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6471
I20260925 16:27:29.690106 7801 tablet_replica.cc:330] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: stopping tablet replica
I20260925 16:27:29.690675 7801 raft_consensus.cc:2229] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:29.691238 7801 raft_consensus.cc:2258] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:29.694957 7801 ts_tablet_manager.cc:1905] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:27:29.698313 7628 connection.cc:535] server connection from 127.0.0.1:38430 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:27:29.699491 7801 ts_tablet_manager.cc:1918] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 16:27:29.699584 7493 connection.cc:535] server connection from 127.0.0.1:42426 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:27:29.699643 7801 log.cc:1196] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/71f641f157f64dd9aeb99478a8963406
I20260925 16:27:29.700001 7801 ts_tablet_manager.cc:1939] T 71f641f157f64dd9aeb99478a8963406 P 30cce7aad35d402096abc60e5d7a4563: Deleting consensus metadata
W20260925 16:27:29.701854 7358 connection.cc:441] server connection from 127.0.0.1:34182 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34182 (request call id 1) could send its response
I20260925 16:27:29.708243 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.685914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.61:37997)
0925 16:27:29.686050 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.686066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.692039 (+ 5973us) server_negotiation.cc:407] Connection header received
0925 16:27:29.692221 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.692225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.692303 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.692432 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.693608 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.694419 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.699146 (+ 4727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.699530 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.699624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.703294 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.703299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.703303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.703354 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.705899 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.706678 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.706683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.706685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.706788 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.707122 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.707126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.707128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.707381 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.707498 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.707898 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.708058 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:27:29.715977 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.704201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.0.98.60:33391)
0925 16:27:29.704373 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.704378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.704930 (+ 552us) server_negotiation.cc:407] Connection header received
0925 16:27:29.705180 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.705183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.705259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.705322 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.706466 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.707381 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.708874 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.709267 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.709335 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.711515 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.711518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.711521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.711547 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.713792 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.714433 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.714437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.714440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.714552 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.714876 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.714880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.714881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.715123 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.715230 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.715644 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.715799 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:27:29.716261 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.704331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.98.61:37997)
0925 16:27:29.704631 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.704636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.705261 (+ 625us) server_negotiation.cc:407] Connection header received
0925 16:27:29.705414 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.705417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.705486 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.705539 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.706572 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.707381 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.708873 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.709237 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.709323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.711483 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.711489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.711492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.711527 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.713982 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.714587 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.714591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.714593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.714697 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.715062 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.715066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.715068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.715365 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.715490 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.715906 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.716066 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260925 16:27:29.720829 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:29.681553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.98.60:33391)
0925 16:27:29.681718 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:29.681723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:29.709905 (+ 28182us) server_negotiation.cc:407] Connection header received
0925 16:27:29.710036 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:29.710040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:29.710098 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:29.710187 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:29.711309 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.712684 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.714103 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:29.714455 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:29.714545 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:29.716345 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:29.716350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:29.716353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:29.716381 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:29.718743 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.719335 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.719339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.719340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.719443 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:29.719746 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:29.719748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:29.719750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:29.720009 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:29.720134 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:29.720466 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 16:27:29.720639 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 16:27:30.659703 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:30.659031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:39717)
0925 16:27:30.659413 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:30.659550 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
W20260925 16:27:30.660211 7606 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:30.663671 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:30.662999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:36835)
0925 16:27:30.663331 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:30.663450 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":103,"threads_started":1}
I20260925 16:27:30.664218 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:30.663455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:40893)
0925 16:27:30.663977 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:30.664088 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":72,"threads_started":1}
W20260925 16:27:30.664314 7742 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
W20260925 16:27:30.664546 7469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:31.178431 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 538581326ec446cfa4f4f3373754f1fb)
I20260925 16:27:31.178853 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:31.178910 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 538581326ec446cfa4f4f3373754f1fb)
I20260925 16:27:31.179046 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:31.179692 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61:37997)
I20260925 16:27:31.180253 7813 leader_election.cc:288] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60:33391)
I20260925 16:27:31.180877 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.179973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51228)
0925 16:27:31.180606 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.180714 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":50,"thread_start_us":91,"threads_started":1}
I20260925 16:27:31.181212 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.180490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51234)
0925 16:27:31.181020 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.181109 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":64,"threads_started":1}
W20260925 16:27:31.181223 7273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
W20260925 16:27:31.181372 7089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:31.182440 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.182169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51240)
0925 16:27:31.182320 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.182367 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 16:27:31.182579 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.182238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51250)
0925 16:27:31.182374 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.182423 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 16:27:31.182752 7273 leader_election.cc:334] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
W20260925 16:27:31.182868 7089 leader_election.cc:334] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:31.184859 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.180091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.98.61:37997)
0925 16:27:31.180473 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:31.180478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:31.180495 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:27:31.180698 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:31.180702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:31.180755 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:31.180836 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:31.181265 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.182168 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.183627 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.184151 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.184231 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:31.184298 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:31.184638 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 16:27:31.184728 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":94,"threads_started":1}
I20260925 16:27:31.184827 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.179973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37997 (local address 127.0.0.1:37396)
0925 16:27:31.180392 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.180412 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:31.180554 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:31.180977 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:31.180980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:31.180996 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:31.181132 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:31.181139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.182300 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.182303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:31.183441 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:31.183453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.184268 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.184271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:31.184428 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:31.184458 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:27:31.184550 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:27:31.184653 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260925 16:27:31.185161 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.180452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33391 (local address 127.0.0.1:47206)
0925 16:27:31.180719 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.180733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:31.180840 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:31.181281 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:31.181284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:31.181301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:31.181433 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:31.181440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.182544 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.182547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:31.183762 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:31.183771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.184655 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.184659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:31.184753 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:31.184775 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:27:31.184891 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 16:27:31.185047 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260925 16:27:31.185353 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.180845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.0.98.60:33391)
0925 16:27:31.181043 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:31.181047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:31.181058 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:27:31.181109 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:31.181112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:31.181154 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:31.181543 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:31.181645 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.182423 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.183935 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:31.184494 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:31.184638 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:31.184689 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:31.185066 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 16:27:31.185148 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260925 16:27:31.185443 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b408cfaa09c43a683c8fbbabb148db0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb3a70b12add4fa6b8e6351fff12935c" is_pre_election: true
I20260925 16:27:31.185655 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b408cfaa09c43a683c8fbbabb148db0 in term 1.
I20260925 16:27:31.185843 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb3a70b12add4fa6b8e6351fff12935c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b408cfaa09c43a683c8fbbabb148db0" is_pre_election: true
I20260925 16:27:31.186028 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b408cfaa09c43a683c8fbbabb148db0, bb3a70b12add4fa6b8e6351fff12935c; no voters: 538581326ec446cfa4f4f3373754f1fb
I20260925 16:27:31.186110 7299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb3a70b12add4fa6b8e6351fff12935c in term 1.
I20260925 16:27:31.186210 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 16:27:31.186266 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 538581326ec446cfa4f4f3373754f1fb)
I20260925 16:27:31.186313 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:27:31.186484 7091 leader_election.cc:302] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b408cfaa09c43a683c8fbbabb148db0, bb3a70b12add4fa6b8e6351fff12935c; no voters: 538581326ec446cfa4f4f3373754f1fb
I20260925 16:27:31.186728 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 16:27:31.186791 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 538581326ec446cfa4f4f3373754f1fb)
I20260925 16:27:31.186833 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:27:31.188000 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:31.188210 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:31.188531 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 2 election: Requested vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61:37997)
I20260925 16:27:31.188618 7813 leader_election.cc:288] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [CANDIDATE]: Term 2 election: Requested vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), 9b408cfaa09c43a683c8fbbabb148db0 (127.0.98.60:33391)
I20260925 16:27:31.189132 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.188875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51270)
0925 16:27:31.188991 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.189048 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 16:27:31.189314 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.188813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51254)
0925 16:27:31.189061 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.189190 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 16:27:31.189742 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb3a70b12add4fa6b8e6351fff12935c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b408cfaa09c43a683c8fbbabb148db0"
I20260925 16:27:31.189741 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b408cfaa09c43a683c8fbbabb148db0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb3a70b12add4fa6b8e6351fff12935c"
I20260925 16:27:31.190003 7299 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bb3a70b12add4fa6b8e6351fff12935c in current term 2: Already voted for candidate 9b408cfaa09c43a683c8fbbabb148db0 in this term.
I20260925 16:27:31.190035 7111 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9b408cfaa09c43a683c8fbbabb148db0 in current term 2: Already voted for candidate bb3a70b12add4fa6b8e6351fff12935c in this term.
I20260925 16:27:31.190421 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.190051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51286)
0925 16:27:31.190288 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.190347 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260925 16:27:31.190511 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.189672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51272)
0925 16:27:31.190371 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.190433 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613}
W20260925 16:27:31.190517 7273 leader_election.cc:334] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
W20260925 16:27:31.190593 7089 leader_election.cc:334] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:31.190624 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b408cfaa09c43a683c8fbbabb148db0; no voters: 538581326ec446cfa4f4f3373754f1fb, bb3a70b12add4fa6b8e6351fff12935c
I20260925 16:27:31.190649 7089 leader_election.cc:302] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb3a70b12add4fa6b8e6351fff12935c; no voters: 538581326ec446cfa4f4f3373754f1fb, 9b408cfaa09c43a683c8fbbabb148db0
I20260925 16:27:31.190944 7813 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 16:27:31.191099 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 16:27:31.661681 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.661039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:38673)
0925 16:27:31.661448 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.661539 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260925 16:27:31.665889 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.665077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:58617)
0925 16:27:31.665657 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.665738 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":135,"threads_started":1}
I20260925 16:27:31.666167 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:31.665128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:55411)
0925 16:27:31.665826 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:31.665915 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":122,"threads_started":1}
I20260925 16:27:32.663394 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.662684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:52239)
0925 16:27:32.663150 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.663240 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":118,"threads_started":1}
I20260925 16:27:32.667693 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.666925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:35007)
0925 16:27:32.667479 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.667564 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":194,"threads_started":1}
I20260925 16:27:32.668376 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.667499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:34579)
0925 16:27:32.668196 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.668263 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":84,"threads_started":1}
I20260925 16:27:32.713326 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:32.713560 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:32.714433 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61:37997)
I20260925 16:27:32.715212 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b408cfaa09c43a683c8fbbabb148db0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb3a70b12add4fa6b8e6351fff12935c" is_pre_election: true
I20260925 16:27:32.715451 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b408cfaa09c43a683c8fbbabb148db0 in term 2.
I20260925 16:27:32.715485 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.714873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51510)
0925 16:27:32.715267 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.715348 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":151,"threads_started":1}
I20260925 16:27:32.715976 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b408cfaa09c43a683c8fbbabb148db0, bb3a70b12add4fa6b8e6351fff12935c; no voters:
I20260925 16:27:32.716251 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 16:27:32.716320 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:32.716347 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:27:32.716653 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.716332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51512)
0925 16:27:32.716487 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.716564 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 16:27:32.716887 7273 leader_election.cc:334] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:32.717700 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:32.718187 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 3 election: Requested vote from peers 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097), bb3a70b12add4fa6b8e6351fff12935c (127.0.98.61:37997)
I20260925 16:27:32.719022 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.718653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51526)
0925 16:27:32.718850 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.718932 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 16:27:32.718930 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b408cfaa09c43a683c8fbbabb148db0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb3a70b12add4fa6b8e6351fff12935c"
I20260925 16:27:32.719126 7111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:27:32.720014 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.719660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51532)
0925 16:27:32.719836 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.719906 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 16:27:32.720221 7273 leader_election.cc:334] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
I20260925 16:27:32.720849 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b408cfaa09c43a683c8fbbabb148db0 in term 3.
I20260925 16:27:32.721354 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b408cfaa09c43a683c8fbbabb148db0, bb3a70b12add4fa6b8e6351fff12935c; no voters: 538581326ec446cfa4f4f3373754f1fb
I20260925 16:27:32.721612 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:27:32.721935 7829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [term 3 LEADER]: Becoming Leader. State: Replica: 9b408cfaa09c43a683c8fbbabb148db0, State: Running, Role: LEADER
I20260925 16:27:32.722295 7829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } }
I20260925 16:27:32.724401 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b408cfaa09c43a683c8fbbabb148db0. Latest consensus state: current_term: 3 leader_uuid: "9b408cfaa09c43a683c8fbbabb148db0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:32.724604 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:32.724989 7834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:32.727820 7834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:32.728317 7834 catalog_manager.cc:1259] Loaded cluster ID: 3c397eaee7914c7eb7a551bd81a560ba
I20260925 16:27:32.728370 7834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:32.728981 7834 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:32.729377 7834 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Loaded TSK: 0
I20260925 16:27:32.730214 7834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:32.751745 7289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:32.756568 7289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:32.761250 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.760945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51552)
0925 16:27:32.761077 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.761144 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 16:27:32.761438 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [term 3 FOLLOWER]: Refusing update from remote peer 9b408cfaa09c43a683c8fbbabb148db0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 16:27:32.762141 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 16:27:32.762347 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.762045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:51558)
0925 16:27:32.762183 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.762256 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 16:27:32.762473 7273 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 -> Peer 538581326ec446cfa4f4f3373754f1fb (127.0.98.62:34097): Couldn't send request to peer 538581326ec446cfa4f4f3373754f1fb. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 16:27:32.764279 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b408cfaa09c43a683c8fbbabb148db0. Latest consensus state: current_term: 3 leader_uuid: "9b408cfaa09c43a683c8fbbabb148db0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:32.764490 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:32.766371 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9b408cfaa09c43a683c8fbbabb148db0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:32.766822 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:32.767509 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9b408cfaa09c43a683c8fbbabb148db0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "538581326ec446cfa4f4f3373754f1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } peers { permanent_uuid: "bb3a70b12add4fa6b8e6351fff12935c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37997 } attrs { promote: false } } peers { permanent_uuid: "9b408cfaa09c43a683c8fbbabb148db0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33391 } attrs { promote: false } } }
I20260925 16:27:32.767709 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3a70b12add4fa6b8e6351fff12935c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:27:32.780957 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.776414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.98.3:40925)
0925 16:27:32.776708 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:32.776712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:32.776733 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:27:32.776920 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:32.776923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:32.776974 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:32.777082 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:32.777421 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.778200 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.779724 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.780299 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.780450 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.780530 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:32.780719 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 16:27:32.780798 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260925 16:27:32.780946 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.776213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40925 (local address 127.0.0.1:38436)
0925 16:27:32.776414 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.776435 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:32.776613 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:32.777130 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:32.777134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:32.777160 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:32.777270 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.777277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.778339 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.778343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.779561 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.779572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.780467 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.780471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.780593 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.780618 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:27:32.780690 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:27:32.780788 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 16:27:32.782159 7675 tablet_service.cc:1467] Processing CreateTablet for tablet bae5f769cc08456abd265ce76665623f (DEFAULT_TABLE table=test-table [id=10301df9df7f4d8ea51adc448db81840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:32.782618 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae5f769cc08456abd265ce76665623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:32.782617 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.777863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34861 (local address 127.0.0.1:34194)
0925 16:27:32.778211 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.778227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:32.778308 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:32.778760 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:32.778764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:32.778783 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:32.778906 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.778911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.779867 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.779871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.781079 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.781088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.782152 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.782155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.782259 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.782283 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:27:32.782390 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 16:27:32.782486 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":130,"threads_started":1}
I20260925 16:27:32.782689 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.777937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.98.1:34861)
0925 16:27:32.778502 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:32.778506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:32.778519 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:27:32.778570 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:32.778573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:32.778621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:32.778716 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:32.779029 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.779740 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.781270 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.781983 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.782111 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.782200 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:32.782412 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 16:27:32.782523 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":87,"threads_started":1}
I20260925 16:27:32.783782 7402 tablet_service.cc:1467] Processing CreateTablet for tablet bae5f769cc08456abd265ce76665623f (DEFAULT_TABLE table=test-table [id=10301df9df7f4d8ea51adc448db81840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:32.784162 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae5f769cc08456abd265ce76665623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:32.785993 7846 tablet_bootstrap.cc:492] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Bootstrap starting.
I20260925 16:27:32.787034 7846 tablet_bootstrap.cc:654] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:32.787572 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.777105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39793 (local address 127.0.0.1:42430)
0925 16:27:32.777483 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.777499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:32.777584 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:32.778150 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:32.778154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:32.778175 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:32.778305 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.778313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.779323 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.779327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.781608 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.781617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.787051 (+ 5434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.787055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.787166 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.787190 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:27:32.787283 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:27:32.787380 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":176,"threads_started":1}
I20260925 16:27:32.787946 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.777299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.0.98.2:39793)
0925 16:27:32.777602 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:32.777608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:32.777625 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:27:32.777696 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:32.777699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:32.777864 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:32.777985 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:32.778444 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.779211 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.786059 (+ 6848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.786751 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.786863 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.786943 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:32.787368 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 16:27:32.787532 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260925 16:27:32.788358 7847 tablet_bootstrap.cc:492] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563: Bootstrap starting.
I20260925 16:27:32.788676 7539 tablet_service.cc:1467] Processing CreateTablet for tablet bae5f769cc08456abd265ce76665623f (DEFAULT_TABLE table=test-table [id=10301df9df7f4d8ea51adc448db81840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:32.789014 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae5f769cc08456abd265ce76665623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:32.789459 7847 tablet_bootstrap.cc:654] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:32.790776 7846 tablet_bootstrap.cc:492] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: No bootstrap required, opened a new log
I20260925 16:27:32.790898 7846 ts_tablet_manager.cc:1397] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 16:27:32.791364 7847 tablet_bootstrap.cc:492] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563: No bootstrap required, opened a new log
I20260925 16:27:32.791515 7847 ts_tablet_manager.cc:1397] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:32.791448 7846 raft_consensus.cc:348] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.791594 7846 raft_consensus.cc:374] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:32.791622 7846 raft_consensus.cc:729] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1991305fb884eaf93a8ec27104cada9, State: Initialized, Role: FOLLOWER
I20260925 16:27:32.791760 7846 consensus_queue.cc:260] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.792165 7846 ts_tablet_manager.cc:1428] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:32.792207 7847 raft_consensus.cc:348] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.792353 7847 raft_consensus.cc:374] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:32.792384 7847 raft_consensus.cc:729] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cce7aad35d402096abc60e5d7a4563, State: Initialized, Role: FOLLOWER
I20260925 16:27:32.792529 7847 consensus_queue.cc:260] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.792949 7847 ts_tablet_manager.cc:1428] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:32.793263 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.792727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:34117)
0925 16:27:32.793063 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.793161 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260925 16:27:32.793401 7851 tablet_bootstrap.cc:492] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Bootstrap starting.
I20260925 16:27:32.794231 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.793901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:37103)
0925 16:27:32.794107 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.794163 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 16:27:32.794637 7851 tablet_bootstrap.cc:654] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:32.795562 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.795236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:56233)
0925 16:27:32.795406 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.795474 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 16:27:32.796368 7851 tablet_bootstrap.cc:492] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: No bootstrap required, opened a new log
I20260925 16:27:32.796495 7851 ts_tablet_manager.cc:1397] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260925 16:27:32.797127 7851 raft_consensus.cc:348] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.797250 7851 raft_consensus.cc:374] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:32.797281 7851 raft_consensus.cc:729] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3cd4ee37e74cb2950ec70812ae8a6a, State: Initialized, Role: FOLLOWER
I20260925 16:27:32.797434 7851 consensus_queue.cc:260] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.797966 7851 ts_tablet_manager.cc:1428] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:32.798975 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.798669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:48771)
0925 16:27:32.798811 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.798881 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 16:27:32.799320 7468 heartbeater.cc:503] Master 127.0.98.60:33391 was elected leader, sending a full tablet report...
I20260925 16:27:32.800483 7741 heartbeater.cc:503] Master 127.0.98.60:33391 was elected leader, sending a full tablet report...
I20260925 16:27:32.800629 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.800230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:50937)
0925 16:27:32.800426 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.800525 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 16:27:32.802973 7605 heartbeater.cc:503] Master 127.0.98.60:33391 was elected leader, sending a full tablet report...
I20260925 16:27:32.990026 7850 raft_consensus.cc:482] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:32.990278 7850 raft_consensus.cc:504] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:32.991724 7850 leader_election.cc:288] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861), fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793)
I20260925 16:27:32.998255 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.992292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41675 (local address 127.0.98.2:39793)
0925 16:27:32.992453 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:32.992458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:32.992577 (+ 119us) server_negotiation.cc:407] Connection header received
0925 16:27:32.992774 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:32.992777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:32.992838 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:32.992914 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:32.993378 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.994301 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.996584 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.997287 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.997455 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.997534 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:32.998002 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 16:27:32.998095 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:27:32.998255 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.992405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46711 (local address 127.0.98.1:34861)
0925 16:27:32.992548 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:32.992553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:32.992765 (+ 212us) server_negotiation.cc:407] Connection header received
0925 16:27:32.992877 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:32.992880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:32.992932 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:32.992998 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:32.993378 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.994300 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.996584 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.997287 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.997455 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.997534 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:32.997972 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 16:27:32.998095 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:27:32.998294 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.992252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34861 (local address 127.0.98.3:46711)
0925 16:27:32.992635 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.992646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:32.992788 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:32.993012 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:32.993015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:32.993027 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:32.993152 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.993157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.994514 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.994518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.996317 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.996344 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.997514 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.997519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.997715 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.997798 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 16:27:32.997893 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 16:27:32.998068 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":237,"thread_start_us":172,"threads_started":1}
I20260925 16:27:32.998294 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:32.992203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39793 (local address 127.0.98.3:41675)
0925 16:27:32.992453 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:32.992471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:32.992619 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:32.992934 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:32.992936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:32.992951 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:32.993147 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.993156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.994514 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.994518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.996317 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:32.996344 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:32.997514 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:32.997519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:32.997828 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:32.997849 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:27:32.997918 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 16:27:32.998068 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":210}
I20260925 16:27:32.999136 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae5f769cc08456abd265ce76665623f" candidate_uuid: "a1991305fb884eaf93a8ec27104cada9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cce7aad35d402096abc60e5d7a4563" is_pre_election: true
I20260925 16:27:32.999480 7422 raft_consensus.cc:2454] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1991305fb884eaf93a8ec27104cada9 in term 0.
I20260925 16:27:32.999573 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae5f769cc08456abd265ce76665623f" candidate_uuid: "a1991305fb884eaf93a8ec27104cada9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" is_pre_election: true
I20260925 16:27:32.999856 7558 raft_consensus.cc:2454] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1991305fb884eaf93a8ec27104cada9 in term 0.
I20260925 16:27:33.000075 7628 leader_election.cc:302] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cce7aad35d402096abc60e5d7a4563, a1991305fb884eaf93a8ec27104cada9; no voters:
I20260925 16:27:33.000522 7850 raft_consensus.cc:2781] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:33.000617 7850 raft_consensus.cc:482] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:33.000645 7850 raft_consensus.cc:3037] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:33.002038 7850 raft_consensus.cc:504] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:33.002482 7850 leader_election.cc:288] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [CANDIDATE]: Term 1 election: Requested vote from peers 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861), fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793)
I20260925 16:27:33.003091 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae5f769cc08456abd265ce76665623f" candidate_uuid: "a1991305fb884eaf93a8ec27104cada9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a"
I20260925 16:27:33.003237 7558 raft_consensus.cc:3037] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:33.003154 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae5f769cc08456abd265ce76665623f" candidate_uuid: "a1991305fb884eaf93a8ec27104cada9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cce7aad35d402096abc60e5d7a4563"
I20260925 16:27:33.003367 7422 raft_consensus.cc:3037] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:33.004503 7558 raft_consensus.cc:2454] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1991305fb884eaf93a8ec27104cada9 in term 1.
I20260925 16:27:33.004971 7422 raft_consensus.cc:2454] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1991305fb884eaf93a8ec27104cada9 in term 1.
I20260925 16:27:33.005019 7631 leader_election.cc:302] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1991305fb884eaf93a8ec27104cada9, fe3cd4ee37e74cb2950ec70812ae8a6a; no voters:
I20260925 16:27:33.005507 7850 raft_consensus.cc:2781] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:33.005860 7850 raft_consensus.cc:686] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 1 LEADER]: Becoming Leader. State: Replica: a1991305fb884eaf93a8ec27104cada9, State: Running, Role: LEADER
I20260925 16:27:33.006273 7850 consensus_queue.cc:237] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } }
I20260925 16:27:33.006865 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.006523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:44303)
0925 16:27:33.006663 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.006753 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 16:27:33.009588 7289 catalog_manager.cc:5495] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1991305fb884eaf93a8ec27104cada9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a1991305fb884eaf93a8ec27104cada9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1991305fb884eaf93a8ec27104cada9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:33.032787 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.027973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.98.3:40925)
0925 16:27:33.028188 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:33.028192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:33.028216 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:27:33.028594 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:33.028598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:33.028687 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:33.028815 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:33.029175 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:33.029748 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:33.030899 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:33.031148 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:33.031257 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:33.031609 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:33.032263 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:33.032384 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:33.032530 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 16:27:33.032601 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 16:27:33.036985 7558 raft_consensus.cc:1261] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Refusing update from remote peer a1991305fb884eaf93a8ec27104cada9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:33.037339 7422 raft_consensus.cc:1261] T bae5f769cc08456abd265ce76665623f P 30cce7aad35d402096abc60e5d7a4563 [term 1 FOLLOWER]: Refusing update from remote peer a1991305fb884eaf93a8ec27104cada9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:33.037700 7850 consensus_queue.cc:1035] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3cd4ee37e74cb2950ec70812ae8a6a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:33.038110 7856 consensus_queue.cc:1035] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cce7aad35d402096abc60e5d7a4563" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:33.041059 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.040198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:60507)
0925 16:27:33.040855 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.040933 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567}
I20260925 16:27:33.041148 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.040645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:59537)
0925 16:27:33.041022 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.041073 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308}
I20260925 16:27:33.045310 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.044921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:55675)
0925 16:27:33.045038 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.045179 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 16:27:33.045998 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.045638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:46953)
0925 16:27:33.045849 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.045911 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 16:27:33.047369 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.047063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:57389)
0925 16:27:33.047217 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.047283 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 16:27:33.055398 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.054872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:40291)
0925 16:27:33.055169 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.055248 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260925 16:27:33.055552 7289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:33.055768 7289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:33.062001 7289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b408cfaa09c43a683c8fbbabb148db0: Sending DeleteTablet for 3 replicas of tablet bae5f769cc08456abd265ce76665623f
I20260925 16:27:33.063084 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet bae5f769cc08456abd265ce76665623f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34194
I20260925 16:27:33.063894 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet bae5f769cc08456abd265ce76665623f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38436
I20260925 16:27:33.064119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 7249
I20260925 16:27:33.066074 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet bae5f769cc08456abd265ce76665623f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42430
I20260925 16:27:33.066685 7871 tablet_replica.cc:330] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: stopping tablet replica
I20260925 16:27:33.066915 7871 raft_consensus.cc:2229] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:33.067302 7871 raft_consensus.cc:2258] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:33.068463 7871 ts_tablet_manager.cc:1905] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:33.070130 7872 tablet_replica.cc:330] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: stopping tablet replica
I20260925 16:27:33.070446 7872 raft_consensus.cc:2229] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:33.070766 7872 raft_consensus.cc:2258] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:33.072014 7871 ts_tablet_manager.cc:1918] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:33.072108 7872 ts_tablet_manager.cc:1905] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:33.072237 7871 log.cc:1196] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/bae5f769cc08456abd265ce76665623f
I20260925 16:27:33.072696 7871 ts_tablet_manager.cc:1939] T bae5f769cc08456abd265ce76665623f P a1991305fb884eaf93a8ec27104cada9: Deleting consensus metadata
I20260925 16:27:33.074323 7274 catalog_manager.cc:4841] TS a1991305fb884eaf93a8ec27104cada9 (127.0.98.3:40925): tablet bae5f769cc08456abd265ce76665623f (table test-table [id=10301df9df7f4d8ea51adc448db81840]) successfully deleted
I20260925 16:27:33.075685 7872 ts_tablet_manager.cc:1918] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:33.075796 7872 log.cc:1196] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/bae5f769cc08456abd265ce76665623f
I20260925 16:27:33.076118 7872 ts_tablet_manager.cc:1939] T bae5f769cc08456abd265ce76665623f P fe3cd4ee37e74cb2950ec70812ae8a6a: Deleting consensus metadata
I20260925 16:27:33.077329 7276 catalog_manager.cc:4841] TS fe3cd4ee37e74cb2950ec70812ae8a6a (127.0.98.2:39793): tablet bae5f769cc08456abd265ce76665623f (table test-table [id=10301df9df7f4d8ea51adc448db81840]) successfully deleted
I20260925 16:27:33.078382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 7476
I20260925 16:27:33.078783 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:33.078376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34861 (local address 127.0.0.1:34208)
0925 16:27:33.078593 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:33.078669 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 16:27:33.079365 7273 catalog_manager.cc:4584] TS 30cce7aad35d402096abc60e5d7a4563 (127.0.98.1:34861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bae5f769cc08456abd265ce76665623f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34861: connect: Connection refused (error 111)
I20260925 16:27:33.086807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 7612
I20260925 16:27:33.093828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 6967
I20260925 16:27:33.101291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 7069
2026-09-25T16:27:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7864 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:33 dist-test-slave-31kt krb5kdc[7875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:33 dist-test-slave-31kt krb5kdc[7875](info): set up 2 sockets
Sep 25 16:27:33 dist-test-slave-31kt krb5kdc[7875](info): commencing operation
krb5kdc: starting...
W20260925 16:27:34.094904 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:34 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 16:27:34.238744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34793
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:34.383705 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:34.384159 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:34.384220 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:34.389644 7888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:34.389873 7888 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 16:27:34.389909 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:34.389933 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:34.389958 7888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:34.389989 7888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:34.397428 7888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34793
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:34.399173 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:34.400663 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:34.413029 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:34.413033 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:34.413034 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:34.413103 7888 server_base.cc:1034] running on GCE node
I20260925 16:27:34.414448 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:34.416267 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:34.417515 7888 hybrid_clock.cc:648] HybridClock initialized: now 1790353654417474 us; error 65 us; skew 500 ppm
Sep 25 16:27:34 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:34.425856 7888 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 16:27:34.427301 7888 webserver.cc:460] Webserver started at http://127.0.98.62:42585/ using document root <none> and password file <none>
I20260925 16:27:34.428094 7888 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:34.428182 7888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:34.428419 7888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:34.431346 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "d3e5e49ea5594f0388aa2da8d89f3289"
format_stamp: "Formatted at 2026-09-25 16:27:34 on dist-test-slave-31kt"
server_key: "ad8d913574ee49638164e3d018c3d08e"
server_key_iv: "98ee0acf1c6d8c501283285fda1a6e3f"
server_key_version: "encryptionkey@0"
I20260925 16:27:34.432090 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3e5e49ea5594f0388aa2da8d89f3289"
format_stamp: "Formatted at 2026-09-25 16:27:34 on dist-test-slave-31kt"
server_key: "ad8d913574ee49638164e3d018c3d08e"
server_key_iv: "98ee0acf1c6d8c501283285fda1a6e3f"
server_key_version: "encryptionkey@0"
I20260925 16:27:34.437495 7888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:27:34.441319 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:34.443434 7888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:34.443646 7888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "d3e5e49ea5594f0388aa2da8d89f3289"
format_stamp: "Formatted at 2026-09-25 16:27:34 on dist-test-slave-31kt"
server_key: "ad8d913574ee49638164e3d018c3d08e"
server_key_iv: "98ee0acf1c6d8c501283285fda1a6e3f"
server_key_version: "encryptionkey@0"
I20260925 16:27:34.443814 7888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:34.463837 7888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:34.467809 7888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:34.468001 7888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:34.478456 7888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34793
I20260925 16:27:34.478662 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34793 every 8 connection(s)
I20260925 16:27:34.480022 7888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:34.484969 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:34.489202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 7888
I20260925 16:27:34.489349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:34.489717 392 external_mini_cluster.cc:1428] Setting key 87a7bb1f5ec46349ab4ec9fa32e9faa4
I20260925 16:27:34.492550 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Bootstrap starting.
I20260925 16:27:34.495586 7966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:34.496433 7966 log.cc:824] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:34.499292 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: No bootstrap required, opened a new log
Sep 25 16:27:34 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:34.503937 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34793 } }
I20260925 16:27:34.504298 7966 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:34.504354 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e5e49ea5594f0388aa2da8d89f3289, State: Initialized, Role: FOLLOWER
I20260925 16:27:34.504892 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34793 } }
I20260925 16:27:34.505052 7966 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:34.505126 7966 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:34.505237 7966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:34.506624 7966 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34793 } }
I20260925 16:27:34.507131 7966 leader_election.cc:302] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3e5e49ea5594f0388aa2da8d89f3289; no voters:
I20260925 16:27:34.507537 7966 leader_election.cc:288] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:34.507759 7971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:34.508124 7971 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [term 1 LEADER]: Becoming Leader. State: Replica: d3e5e49ea5594f0388aa2da8d89f3289, State: Running, Role: LEADER
I20260925 16:27:34.508736 7971 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34793 } }
I20260925 16:27:34.509410 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:34.509477 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:34.491915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.0.98.62:34793)
0925 16:27:34.492549 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:34.492571 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:34.492621 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:27:34.493523 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:34.493548 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:34.494127 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:34.494574 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:34.495632 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.496996 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.498292 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.498874 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.499024 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:34.502501 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:34.502509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:34.502536 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:34.502588 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:34.504785 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:34.505468 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:34.505477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:34.505484 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:34.505673 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:34.506025 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:34.506029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:34.506031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:34.506679 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:34.506936 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:34.507274 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 16:27:34.507739 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":178,"threads_started":1}
I20260925 16:27:34.511104 7973 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3e5e49ea5594f0388aa2da8d89f3289. Latest consensus state: current_term: 1 leader_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34793 } } }
I20260925 16:27:34.511307 7973 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:34.511672 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e5e49ea5594f0388aa2da8d89f3289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34793 } } }
I20260925 16:27:34.511765 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289 [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:34.515127 7986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:34.515229 7986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:34.515318 7983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:34.516105 7983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:34.524569 7983 catalog_manager.cc:1347] Generated new cluster ID: cb495569c0f14d7f8bfd796a45fa749b
I20260925 16:27:34.524704 7983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:34.534564 7983 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:34.535795 7983 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:34.548597 7983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Generated new TSK 0
I20260925 16:27:34.549558 7983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 16:27:34.582315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34793
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:34.742702 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:34.743037 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:34.743103 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:34.748289 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:34.748418 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:34.748481 7992 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:34.748531 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:34.760668 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:34.762051 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:34.763288 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:34.782037 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:34.782485 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:34.782114 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:34.784927 7992 server_base.cc:1034] running on GCE node
I20260925 16:27:34.786060 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:34.787511 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:34.789027 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790353654788926 us; error 95 us; skew 500 ppm
Sep 25 16:27:34 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:34.793540 7992 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:27:34.795076 7992 webserver.cc:460] Webserver started at http://127.0.98.1:32891/ using document root <none> and password file <none>
I20260925 16:27:34.795881 7992 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:34.795977 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:34.796245 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:34.799463 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "8969e5a2c4404813b1a48193955a81e0"
format_stamp: "Formatted at 2026-09-25 16:27:34 on dist-test-slave-31kt"
server_key: "4ad7ab52542cda93a6fc452b06c7c245"
server_key_iv: "f13cf8219b41eb6d935c8010d6eff0a7"
server_key_version: "encryptionkey@0"
I20260925 16:27:34.800309 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8969e5a2c4404813b1a48193955a81e0"
format_stamp: "Formatted at 2026-09-25 16:27:34 on dist-test-slave-31kt"
server_key: "4ad7ab52542cda93a6fc452b06c7c245"
server_key_iv: "f13cf8219b41eb6d935c8010d6eff0a7"
server_key_version: "encryptionkey@0"
I20260925 16:27:34.806910 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 16:27:34.812106 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:34.813928 7992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:27:34.814096 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "8969e5a2c4404813b1a48193955a81e0"
format_stamp: "Formatted at 2026-09-25 16:27:34 on dist-test-slave-31kt"
server_key: "4ad7ab52542cda93a6fc452b06c7c245"
server_key_iv: "f13cf8219b41eb6d935c8010d6eff0a7"
server_key_version: "encryptionkey@0"
I20260925 16:27:34.814244 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:34.865602 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:34.872841 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:34.873977 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:34.876612 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:34.876695 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:34.876755 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:34.876770 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:34.891386 7992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44165
I20260925 16:27:34.892373 8122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44165 every 8 connection(s)
I20260925 16:27:34.892860 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 16:27:34 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:34.895228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 7992
I20260925 16:27:34.895380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:34.895788 392 external_mini_cluster.cc:1428] Setting key 60fd81787e06f0b98cd66f012cede86f
I20260925 16:27:34.907052 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:34.878353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50321 (local address 127.0.98.62:34793)
0925 16:27:34.878569 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:34.878576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:34.880025 (+ 1449us) server_negotiation.cc:407] Connection header received
0925 16:27:34.881995 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:34.881999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:34.882083 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:34.882167 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:34.885505 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.887362 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.890933 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.891291 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.891420 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:34.895848 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:34.895855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:34.895862 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:34.895909 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:34.897664 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:34.898831 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:34.898835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:34.898836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:34.898970 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:34.903912 (+ 4942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:34.903918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:34.903920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:34.904218 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:34.904337 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:34.906714 (+ 2377us) server_negotiation.cc:299] Negotiation successful
0925 16:27:34.906890 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:27:34.908942 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:34.878867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.98.1:50321)
0925 16:27:34.879742 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:34.879850 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:34.881578 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:34.882438 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:34.882450 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:34.883027 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:34.883800 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:34.883827 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.887571 (+ 3744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.887576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:34.890715 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:34.890737 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.891525 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.891532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:34.891817 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:34.893081 (+ 1264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:34.893113 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:34.895486 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:34.897882 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:34.897899 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:34.897929 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:34.898639 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:34.899148 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:34.899152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:34.899154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:34.899444 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:34.905266 (+ 5822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:34.905282 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:34.906082 (+ 800us) client_negotiation.cc:769] Sending connection context
0925 16:27:34.906517 (+ 435us) client_negotiation.cc:240] Negotiation successful
0925 16:27:34.907244 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":193,"threads_started":1}
I20260925 16:27:34.911305 8123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34793
I20260925 16:27:34.911726 8123 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:34.912793 8123 heartbeater.cc:511] Master 127.0.98.62:34793 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 16:27:34.916775 7919 ts_manager.cc:194] Registered new tserver with Master: 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165)
I20260925 16:27:34.918807 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50321
I20260925 16:27:34.928339 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:34.916820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.98.1:38189)
0925 16:27:34.917369 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:34.917389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:34.917526 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:34.917982 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:34.917985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:34.918291 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:34.918451 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:34.918459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.920091 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.920096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:34.920904 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:34.920917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.922133 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.922138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:34.922259 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:34.922887 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:34.922901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:34.923515 (+ 614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:34.926004 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:34.926010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:34.926013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:34.926321 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:34.926827 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:34.926830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:34.926831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:34.926946 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:34.927690 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:34.927693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:34.927832 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 16:27:34.927947 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 16:27:34.928108 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":12,"thread_start_us":133,"threads_started":1}
I20260925 16:27:34.928886 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:34.917402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38189 (local address 127.0.98.62:34793)
0925 16:27:34.917582 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:34.917586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:34.917609 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:27:34.917670 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:34.917673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:34.917734 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:34.917917 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:34.919125 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.919929 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.921070 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:34.921951 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:34.922098 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:34.923723 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:34.923727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:34.923729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:34.923758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:34.925686 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:34.926526 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:34.926530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:34.926531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:34.926645 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:34.927094 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:34.927097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:34.927098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:34.927532 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:34.927632 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:34.928079 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 16:27:34.928242 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 16:27:34.949223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34793
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:35.144248 8132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:35.145295 8132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:35.146201 8132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:35.153051 8132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:35.153203 8132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:35.153255 8132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:35.153294 8132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:35.163844 8132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:35.165479 8132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:35.166985 8132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:35.186599 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:35.186573 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:35.186581 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:35.187700 8132 server_base.cc:1034] running on GCE node
I20260925 16:27:35.188500 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:35.189465 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:35.192879 8132 hybrid_clock.cc:648] HybridClock initialized: now 1790353655192808 us; error 92 us; skew 500 ppm
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:35.196926 8132 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:27:35.198514 8132 webserver.cc:460] Webserver started at http://127.0.98.2:33781/ using document root <none> and password file <none>
I20260925 16:27:35.199331 8132 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:35.199581 8132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:35.200073 8132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:35.202870 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "e8d63b75f871432e94695f723a5fd2ce"
format_stamp: "Formatted at 2026-09-25 16:27:35 on dist-test-slave-31kt"
server_key: "b8f4b0312cb027b6fb9fda44e0e176fc"
server_key_iv: "abf756362b1996bf748921aec0c29c7b"
server_key_version: "encryptionkey@0"
I20260925 16:27:35.203570 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e8d63b75f871432e94695f723a5fd2ce"
format_stamp: "Formatted at 2026-09-25 16:27:35 on dist-test-slave-31kt"
server_key: "b8f4b0312cb027b6fb9fda44e0e176fc"
server_key_iv: "abf756362b1996bf748921aec0c29c7b"
server_key_version: "encryptionkey@0"
I20260925 16:27:35.208884 8132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:27:35.213551 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:35.216135 8132 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:27:35.216389 8132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "e8d63b75f871432e94695f723a5fd2ce"
format_stamp: "Formatted at 2026-09-25 16:27:35 on dist-test-slave-31kt"
server_key: "b8f4b0312cb027b6fb9fda44e0e176fc"
server_key_iv: "abf756362b1996bf748921aec0c29c7b"
server_key_version: "encryptionkey@0"
I20260925 16:27:35.216998 8132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:35.233983 8132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:35.238366 8132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:35.238585 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:35.240469 8132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:35.240537 8132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:35.240600 8132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:35.240624 8132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:35.252723 8132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34553
I20260925 16:27:35.252888 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34553 every 8 connection(s)
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:35.254460 8132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:35.259994 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.242188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50719 (local address 127.0.98.62:34793)
0925 16:27:35.242363 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.242367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.243928 (+ 1561us) server_negotiation.cc:407] Connection header received
0925 16:27:35.245471 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.245476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.245556 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.245650 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:35.247908 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.248790 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.250624 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.250975 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.251102 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.254780 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:35.254785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:35.254788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:35.254824 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:35.256625 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.257534 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.257538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.257539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.257698 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.258299 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.258302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.258304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.258598 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:35.258696 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.259648 (+ 952us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.259826 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:27:35.261207 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.242795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.98.2:50719)
0925 16:27:35.243638 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.243741 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.245188 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.246075 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.246088 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.246713 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:35.247607 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.247637 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.248970 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.248974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.250443 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.250470 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.251175 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.251180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.251422 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.252205 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:35.252226 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:35.254498 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:35.256856 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.256864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.256883 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.257324 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.257885 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.257889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.257891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.258156 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.258819 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:35.258826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:35.259223 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.259548 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.259978 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":124,"threads_started":1}
I20260925 16:27:35.263191 8263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34793
I20260925 16:27:35.263827 8263 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:35.263957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 8132
I20260925 16:27:35.264132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:35.264595 392 external_mini_cluster.cc:1428] Setting key 92de9a1b069a0d9cd1b5f06ecacb5cd6
I20260925 16:27:35.264863 8263 heartbeater.cc:511] Master 127.0.98.62:34793 requested a full tablet report, sending...
I20260925 16:27:35.266420 7919 ts_manager.cc:194] Registered new tserver with Master: e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553)
I20260925 16:27:35.267408 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50719
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:27:35.276973 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.266622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.98.2:46903)
0925 16:27:35.267259 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.267278 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.267428 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.267910 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.267913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.268239 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:35.268392 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.268399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.269506 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.269512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.270406 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.270420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.271035 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.271038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.271120 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.271694 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:35.271706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:35.272129 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:35.274626 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.274630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.274633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.274913 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.275352 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.275357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.275358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.275534 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.276311 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:35.276314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:35.276448 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.276566 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.276780 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":183,"threads_started":1}
I20260925 16:27:35.277069 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.266765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46903 (local address 127.0.98.62:34793)
0925 16:27:35.267017 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.267023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.267421 (+ 398us) server_negotiation.cc:407] Connection header received
0925 16:27:35.267581 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.267584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.267654 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.267738 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:35.268524 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.269300 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.270562 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.270901 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.271015 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.272313 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:35.272318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:35.272322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:35.272358 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:35.274431 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.275035 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.275040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.275042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.275156 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.275677 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.275681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.275683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.276109 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:35.276248 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.276683 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.276854 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 16:27:35.298511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34793
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:35.435086 8272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:35.435426 8272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:35.435473 8272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:35.440021 8272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:35.440111 8272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:35.440153 8272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 16:27:35.440188 8272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:35.448716 8272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:35.450142 8272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:35.451345 8272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:35.463527 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:35.463606 8277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:35.463639 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:35.463620 8272 server_base.cc:1034] running on GCE node
I20260925 16:27:35.464589 8272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:35.465508 8272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:35.466759 8272 hybrid_clock.cc:648] HybridClock initialized: now 1790353655466748 us; error 59 us; skew 500 ppm
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:35.470903 8272 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:27:35.472766 8272 webserver.cc:460] Webserver started at http://127.0.98.3:39845/ using document root <none> and password file <none>
I20260925 16:27:35.473657 8272 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:35.473801 8272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:35.474103 8272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:35.477103 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
format_stamp: "Formatted at 2026-09-25 16:27:35 on dist-test-slave-31kt"
server_key: "42989f88723762141860660fb6a75f6b"
server_key_iv: "0b0bd710590d65ed68d2cc4bf477098a"
server_key_version: "encryptionkey@0"
I20260925 16:27:35.478359 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
format_stamp: "Formatted at 2026-09-25 16:27:35 on dist-test-slave-31kt"
server_key: "42989f88723762141860660fb6a75f6b"
server_key_iv: "0b0bd710590d65ed68d2cc4bf477098a"
server_key_version: "encryptionkey@0"
I20260925 16:27:35.484539 8272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 16:27:35.488749 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:35.490907 8272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:27:35.491097 8272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
format_stamp: "Formatted at 2026-09-25 16:27:35 on dist-test-slave-31kt"
server_key: "42989f88723762141860660fb6a75f6b"
server_key_iv: "0b0bd710590d65ed68d2cc4bf477098a"
server_key_version: "encryptionkey@0"
I20260925 16:27:35.491266 8272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:35.513278 8272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:35.517011 8272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:35.517189 8272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:35.519006 8272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:35.519093 8272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:35.519148 8272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:35.519186 8272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:35.531704 8272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40383
I20260925 16:27:35.531777 8402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40383 every 8 connection(s)
I20260925 16:27:35.532768 8272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:35.536140 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.520448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45761 (local address 127.0.98.62:34793)
0925 16:27:35.520655 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.520661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.521953 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 16:27:35.523607 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.523613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.523708 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.523833 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:35.525699 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.526357 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.527834 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.528072 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.528164 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.531478 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:35.531483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:35.531487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:35.531529 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:35.533289 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.534115 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.534120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.534122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.534275 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.534742 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.534746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.534755 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.534993 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:35.535091 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.535838 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.535994 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 16:27:35.537267 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.520956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.98.3:45761)
0925 16:27:35.521680 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.521804 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.523279 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.524165 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.524177 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.524652 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:35.525488 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.525513 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.526492 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.526496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.527701 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.527713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.528235 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.528240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.528466 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.529178 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:35.529202 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:35.531160 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:35.533480 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.533487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.533511 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.533962 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.534403 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.534406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.534408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.534580 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.535117 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:35.535125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:35.535476 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.535687 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.536069 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":175,"threads_started":1}
I20260925 16:27:35.538816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 8272
I20260925 16:27:35.538970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:35.539088 8403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34793
I20260925 16:27:35.539252 8403 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:35.539397 392 external_mini_cluster.cc:1428] Setting key 68b2b5a2581d483e324a4c259c8d7541
I20260925 16:27:35.540037 8403 heartbeater.cc:511] Master 127.0.98.62:34793 requested a full tablet report, sending...
I20260925 16:27:35.541815 7920 ts_manager.cc:194] Registered new tserver with Master: e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:35.542635 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45761
I20260925 16:27:35.550679 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.541997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.98.3:46575)
0925 16:27:35.542534 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.542549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.542674 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.542917 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.542920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.543152 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:35.543281 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.543290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.544062 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.544066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.544883 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.544899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.545570 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.545574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.545739 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.546442 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:35.546458 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:35.546872 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:35.548666 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.548670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.548672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.548987 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.549406 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:35.549410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:35.549412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:35.549564 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:35.550098 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:35.550100 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:35.550229 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.550334 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.550503 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":183,"threads_started":1}
I20260925 16:27:35.550688 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.541965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46575 (local address 127.0.98.62:34793)
0925 16:27:35.542103 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.542107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.542614 (+ 507us) server_negotiation.cc:407] Connection header received
0925 16:27:35.542765 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.542768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.542821 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.542886 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:35.543397 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.543938 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.545149 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.545427 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.545536 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.547081 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:35.547084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:35.547087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:35.547114 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:35.548527 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.549122 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.549127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.549129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.549242 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.549713 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.549716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.549718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.549971 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:35.550079 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.550395 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.550526 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:27:35.555157 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:35.582360 7969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:35.575312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.0.98.62:34793)
0925 16:27:35.575528 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.575534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.575551 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:27:35.575738 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.575741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.575786 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.575859 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:35.576640 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.577186 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.578089 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.578382 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.578478 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.582190 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36968: BlockingRecv error: recv got EOF from 127.0.0.1:36968 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 16:27:35 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:35.593315 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.584405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.0.98.62:34793)
0925 16:27:35.584604 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.584609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.584734 (+ 125us) server_negotiation.cc:407] Connection header received
0925 16:27:35.584890 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.584894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.584943 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.585012 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:35.585744 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.586331 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.587162 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.587386 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.587477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.590174 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:35.590178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:35.590180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:35.590216 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:35.591619 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.592110 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.592113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.592115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.592204 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:35.592450 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:35.592453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:35.592455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:35.592659 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:35.592783 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.593013 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.593148 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 16:27:35.597957 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:35.602135 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:35.621307 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.614508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.0.98.3:40383)
0925 16:27:35.615874 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.615884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.615909 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:27:35.615985 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.615989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.616177 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.616335 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:35.616748 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.617601 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.619528 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.620400 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.620553 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.620748 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.620995 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.621126 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":162,"threads_started":1}
I20260925 16:27:35.621387 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.614262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40383 (local address 127.0.0.1:44946)
0925 16:27:35.614831 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.614885 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.615037 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.616344 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.616350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.616390 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:35.616596 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.616605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.617723 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.617727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.619264 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.619275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.620583 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.620587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.620733 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.620794 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.620946 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.621139 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":3,"spinlock_wait_cycles":3328,"thread_start_us":134,"threads_started":1}
I20260925 16:27:35.621661 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.615343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.98.1:44165)
0925 16:27:35.615790 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.615798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.616799 (+ 1001us) server_negotiation.cc:407] Connection header received
0925 16:27:35.616872 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.616875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.617050 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.617185 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:35.617417 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.618474 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.619888 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.620668 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.620780 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.620949 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.621382 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.621521 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":106,"threads_started":1}
I20260925 16:27:35.621788 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.614537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34553 (local address 127.0.0.1:50732)
0925 16:27:35.615494 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.615508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.615599 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.616356 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.616360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.616394 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:35.616609 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.616616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.618925 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.618936 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.619884 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.619895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.621279 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.621284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.621412 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.621443 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.621517 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.621617 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":64,"spinlock_wait_cycles":3456,"thread_start_us":134,"threads_started":1}
I20260925 16:27:35.622030 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.615132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44165 (local address 127.0.0.1:41244)
0925 16:27:35.616616 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.616630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.616712 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.617177 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.617180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.617196 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:35.617305 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.617312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.618614 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.618618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.619736 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.619746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.621013 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.621016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.621127 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.621149 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.621216 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.621882 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260925 16:27:35.622401 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.615348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.0.98.2:34553)
0925 16:27:35.615853 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.615865 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.615890 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:27:35.615977 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.615982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.616199 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.616373 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:35.617780 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.618808 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.620067 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.620911 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.621021 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.621202 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.622060 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.622234 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":158,"threads_started":1}
I20260925 16:27:35.625524 8337 tablet_service.cc:1467] Processing CreateTablet for tablet c865b059523a4445a56182fb5c7cea64 (DEFAULT_TABLE table=test-table [id=bf8ae102945b4a4a9df9bf5403099892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:35.626678 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c865b059523a4445a56182fb5c7cea64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:35.626499 8057 tablet_service.cc:1467] Processing CreateTablet for tablet c865b059523a4445a56182fb5c7cea64 (DEFAULT_TABLE table=test-table [id=bf8ae102945b4a4a9df9bf5403099892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:35.626499 8197 tablet_service.cc:1467] Processing CreateTablet for tablet c865b059523a4445a56182fb5c7cea64 (DEFAULT_TABLE table=test-table [id=bf8ae102945b4a4a9df9bf5403099892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:35.627806 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c865b059523a4445a56182fb5c7cea64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:35.627806 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c865b059523a4445a56182fb5c7cea64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:35.634692 8432 tablet_bootstrap.cc:492] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Bootstrap starting.
I20260925 16:27:35.637205 8432 tablet_bootstrap.cc:654] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:35.638211 8434 tablet_bootstrap.cc:492] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Bootstrap starting.
I20260925 16:27:35.638875 8432 log.cc:824] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:35.637699 8433 tablet_bootstrap.cc:492] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Bootstrap starting.
I20260925 16:27:35.641571 8434 tablet_bootstrap.cc:654] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:35.641571 8433 tablet_bootstrap.cc:654] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:35.641863 8432 tablet_bootstrap.cc:492] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: No bootstrap required, opened a new log
I20260925 16:27:35.642153 8432 ts_tablet_manager.cc:1397] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 16:27:35.642596 8434 log.cc:824] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:35.642596 8433 log.cc:824] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:35.645076 8433 tablet_bootstrap.cc:492] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: No bootstrap required, opened a new log
I20260925 16:27:35.645076 8434 tablet_bootstrap.cc:492] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: No bootstrap required, opened a new log
I20260925 16:27:35.645331 8433 ts_tablet_manager.cc:1397] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 16:27:35.645331 8434 ts_tablet_manager.cc:1397] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 16:27:35.646482 8432 raft_consensus.cc:348] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.646919 8432 raft_consensus.cc:374] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:35.646991 8432 raft_consensus.cc:729] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7731dff65c34e8fa1605ebaad6bfe45, State: Initialized, Role: FOLLOWER
I20260925 16:27:35.647707 8432 consensus_queue.cc:260] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.648699 8403 heartbeater.cc:503] Master 127.0.98.62:34793 was elected leader, sending a full tablet report...
I20260925 16:27:35.649302 8432 ts_tablet_manager.cc:1428] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:27:35.649966 8434 raft_consensus.cc:348] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.650146 8433 raft_consensus.cc:348] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.650372 8434 raft_consensus.cc:374] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:35.650372 8433 raft_consensus.cc:374] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:35.650435 8434 raft_consensus.cc:729] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d63b75f871432e94695f723a5fd2ce, State: Initialized, Role: FOLLOWER
I20260925 16:27:35.650435 8433 raft_consensus.cc:729] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Initialized, Role: FOLLOWER
I20260925 16:27:35.651005 8434 consensus_queue.cc:260] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.651005 8433 consensus_queue.cc:260] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.651885 8123 heartbeater.cc:503] Master 127.0.98.62:34793 was elected leader, sending a full tablet report...
I20260925 16:27:35.652248 8433 ts_tablet_manager.cc:1428] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:27:35.652978 8434 ts_tablet_manager.cc:1428] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 16:27:35.653362 8263 heartbeater.cc:503] Master 127.0.98.62:34793 was elected leader, sending a full tablet report...
W20260925 16:27:35.756181 8264 tablet.cc:2369] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:27:35.783898 8404 tablet.cc:2369] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:35.829195 8439 raft_consensus.cc:482] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:35.829438 8439 raft_consensus.cc:504] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.830957 8439 leader_election.cc:288] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383), e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553)
I20260925 16:27:35.836045 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.831531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40789 (local address 127.0.98.2:34553)
0925 16:27:35.831695 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.831700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.831933 (+ 233us) server_negotiation.cc:407] Connection header received
0925 16:27:35.832087 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.832091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.832140 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.832223 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:35.832670 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.833312 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.834718 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.835305 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.835393 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.835460 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.835812 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.835900 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 16:27:35.836102 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.831348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34553 (local address 127.0.98.1:40789)
0925 16:27:35.831800 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.831817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.831969 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.832307 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.832311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.832337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:35.832529 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.832537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.833452 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.833456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.834531 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.834544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.835463 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.835466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.835601 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.835640 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.835745 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.835855 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":117,"threads_started":1}
I20260925 16:27:35.836944 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.831384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40383 (local address 127.0.98.1:48925)
0925 16:27:35.831931 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:35.831948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:35.832107 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:35.832615 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:35.832619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:35.832644 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:35.832801 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.832808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.833848 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.833851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.835036 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:35.835051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.836103 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.836107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:35.836558 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.836595 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 16:27:35.836688 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:27:35.836790 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":138,"threads_started":1}
I20260925 16:27:35.837059 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.831549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48925 (local address 127.0.98.3:40383)
0925 16:27:35.831746 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.831751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.832069 (+ 318us) server_negotiation.cc:407] Connection header received
0925 16:27:35.832311 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.832314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.832376 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.832503 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:35.832955 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.833640 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.835232 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.835918 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.836030 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.836108 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.836759 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.836903 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 16:27:35.837443 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865b059523a4445a56182fb5c7cea64" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce" is_pre_election: true
I20260925 16:27:35.837850 8217 raft_consensus.cc:2454] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:35.838128 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865b059523a4445a56182fb5c7cea64" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" is_pre_election: true
I20260925 16:27:35.838438 8010 leader_election.cc:302] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:35.838508 8357 raft_consensus.cc:2454] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:35.838750 8439 raft_consensus.cc:2781] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:35.838864 8439 raft_consensus.cc:482] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:35.838913 8439 raft_consensus.cc:3037] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:35.840178 8439 raft_consensus.cc:504] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.840564 8439 leader_election.cc:288] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Requested vote from peers e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383), e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553)
I20260925 16:27:35.841080 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865b059523a4445a56182fb5c7cea64" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
I20260925 16:27:35.841292 8357 raft_consensus.cc:3037] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:35.841194 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865b059523a4445a56182fb5c7cea64" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce"
I20260925 16:27:35.841555 8217 raft_consensus.cc:3037] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:35.842921 8217 raft_consensus.cc:2454] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:35.843209 8357 raft_consensus.cc:2454] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:35.843536 8010 leader_election.cc:302] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e7731dff65c34e8fa1605ebaad6bfe45, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:35.844060 8439 raft_consensus.cc:2781] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:35.844390 8439 raft_consensus.cc:686] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Running, Role: LEADER
I20260925 16:27:35.844930 8439 consensus_queue.cc:237] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } }
I20260925 16:27:35.850317 7919 catalog_manager.cc:5495] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8969e5a2c4404813b1a48193955a81e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } health_report { overall_health: HEALTHY } } }
I20260925 16:27:35.869961 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:35.865333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.0.98.1:44165)
0925 16:27:35.865542 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:35.865547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:35.865568 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:27:35.865721 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:35.865723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:35.865834 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:35.865954 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:35.866281 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.866833 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.867936 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:35.868173 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:35.868291 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:35.868696 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:35.869263 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:35.869372 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:35.869591 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 16:27:35.869722 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 16:27:35.878877 8357 raft_consensus.cc:1261] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Refusing update from remote peer 8969e5a2c4404813b1a48193955a81e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:35.878877 8217 raft_consensus.cc:1261] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Refusing update from remote peer 8969e5a2c4404813b1a48193955a81e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:35.879992 8443 consensus_queue.cc:1035] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:35.880290 8439 consensus_queue.cc:1035] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:35.891637 8446 mvcc.cc:205] Tried to move back new op lower bound from 7333288574468456448 to 7333288574345879552. Current Snapshot: MvccSnapshot[applied={T|T < 7333288574468456448}]
I20260925 16:27:35.893122 8448 mvcc.cc:205] Tried to move back new op lower bound from 7333288574468456448 to 7333288574345879552. Current Snapshot: MvccSnapshot[applied={T|T < 7333288574468456448}]
I20260925 16:27:35.898955 8447 mvcc.cc:205] Tried to move back new op lower bound from 7333288574468456448 to 7333288574345879552. Current Snapshot: MvccSnapshot[applied={T|T < 7333288574468456448}]
I20260925 16:27:35.907565 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:35.907845 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:35.911128 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Sending DeleteTablet for 3 replicas of tablet c865b059523a4445a56182fb5c7cea64
I20260925 16:27:35.912668 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet c865b059523a4445a56182fb5c7cea64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260925 16:27:35.912714 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet c865b059523a4445a56182fb5c7cea64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50732
I20260925 16:27:35.912757 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet c865b059523a4445a56182fb5c7cea64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41244
I20260925 16:27:35.913448 8456 tablet_replica.cc:330] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: stopping tablet replica
I20260925 16:27:35.913745 8455 tablet_replica.cc:330] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: stopping tablet replica
I20260925 16:27:35.914074 8456 raft_consensus.cc:2229] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:35.914304 8455 raft_consensus.cc:2229] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:35.914217 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:27:35.914608 8456 raft_consensus.cc:2258] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 16:27:35.914866 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:35.915095 8455 raft_consensus.cc:2258] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:35.915860 8457 tablet_replica.cc:330] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: stopping tablet replica
I20260925 16:27:35.916369 8457 raft_consensus.cc:2229] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:35.916903 8457 raft_consensus.cc:2258] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:35.917028 8456 ts_tablet_manager.cc:1905] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:35.918727 8457 ts_tablet_manager.cc:1905] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:35.921336 8456 ts_tablet_manager.cc:1918] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:35.921532 8456 log.cc:1196] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/c865b059523a4445a56182fb5c7cea64
I20260925 16:27:35.921941 8456 ts_tablet_manager.cc:1939] T c865b059523a4445a56182fb5c7cea64 P e8d63b75f871432e94695f723a5fd2ce: Deleting consensus metadata
I20260925 16:27:35.923561 8457 ts_tablet_manager.cc:1918] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:35.923834 8457 log.cc:1196] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/c865b059523a4445a56182fb5c7cea64
I20260925 16:27:35.924269 8457 ts_tablet_manager.cc:1939] T c865b059523a4445a56182fb5c7cea64 P 8969e5a2c4404813b1a48193955a81e0: Deleting consensus metadata
I20260925 16:27:35.924309 7906 catalog_manager.cc:4841] TS e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553): tablet c865b059523a4445a56182fb5c7cea64 (table test-table [id=bf8ae102945b4a4a9df9bf5403099892]) successfully deleted
I20260925 16:27:35.924289 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9334c628124870a7fe64614cb7e28d (DEFAULT_TABLE table=test-table [id=3b4bb3b776ad4d069fd4555380bb36bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:35.924321 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9334c628124870a7fe64614cb7e28d (DEFAULT_TABLE table=test-table [id=3b4bb3b776ad4d069fd4555380bb36bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:35.924649 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9334c628124870a7fe64614cb7e28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:35.924664 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9334c628124870a7fe64614cb7e28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:35.926321 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9334c628124870a7fe64614cb7e28d (DEFAULT_TABLE table=test-table [id=3b4bb3b776ad4d069fd4555380bb36bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:35.926676 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9334c628124870a7fe64614cb7e28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:35.926954 8455 ts_tablet_manager.cc:1905] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:35.928016 8432 tablet_bootstrap.cc:492] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Bootstrap starting.
I20260925 16:27:35.929244 8432 tablet_bootstrap.cc:654] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:35.929677 8434 tablet_bootstrap.cc:492] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Bootstrap starting.
I20260925 16:27:35.930454 8433 tablet_bootstrap.cc:492] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Bootstrap starting.
I20260925 16:27:35.931282 8434 tablet_bootstrap.cc:654] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:35.931337 8432 tablet_bootstrap.cc:492] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: No bootstrap required, opened a new log
I20260925 16:27:35.931427 8432 ts_tablet_manager.cc:1397] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:35.931704 8433 tablet_bootstrap.cc:654] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:35.932044 8432 raft_consensus.cc:348] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:35.932204 8432 raft_consensus.cc:374] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:35.932236 8432 raft_consensus.cc:729] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7731dff65c34e8fa1605ebaad6bfe45, State: Initialized, Role: FOLLOWER
I20260925 16:27:35.932410 8432 consensus_queue.cc:260] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:35.932722 8432 ts_tablet_manager.cc:1428] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:35.933269 8455 ts_tablet_manager.cc:1918] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:35.933393 8455 log.cc:1196] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/c865b059523a4445a56182fb5c7cea64
I20260925 16:27:35.933785 8455 ts_tablet_manager.cc:1939] T c865b059523a4445a56182fb5c7cea64 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting consensus metadata
I20260925 16:27:35.935539 7907 catalog_manager.cc:4841] TS e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383): tablet c865b059523a4445a56182fb5c7cea64 (table test-table [id=bf8ae102945b4a4a9df9bf5403099892]) successfully deleted
I20260925 16:27:35.935905 7907 catalog_manager.cc:4841] TS 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165): tablet c865b059523a4445a56182fb5c7cea64 (table test-table [id=bf8ae102945b4a4a9df9bf5403099892]) successfully deleted
I20260925 16:27:35.937373 8433 tablet_bootstrap.cc:492] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: No bootstrap required, opened a new log
I20260925 16:27:35.937501 8433 ts_tablet_manager.cc:1397] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 16:27:35.937695 8434 tablet_bootstrap.cc:492] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: No bootstrap required, opened a new log
I20260925 16:27:35.937837 8434 ts_tablet_manager.cc:1397] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 16:27:35.938201 8433 raft_consensus.cc:348] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:35.938359 8433 raft_consensus.cc:374] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:35.938391 8433 raft_consensus.cc:729] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Initialized, Role: FOLLOWER
I20260925 16:27:35.938374 8434 raft_consensus.cc:348] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:35.938484 8434 raft_consensus.cc:374] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:35.938512 8434 raft_consensus.cc:729] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d63b75f871432e94695f723a5fd2ce, State: Initialized, Role: FOLLOWER
I20260925 16:27:35.938561 8433 consensus_queue.cc:260] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:35.938648 8434 consensus_queue.cc:260] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:35.938866 8433 ts_tablet_manager.cc:1428] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:35.938944 8434 ts_tablet_manager.cc:1428] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260925 16:27:36.146546 8124 tablet.cc:2369] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:27:36.242280 8443 raft_consensus.cc:482] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:36.242544 8443 raft_consensus.cc:504] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.243079 8443 leader_election.cc:288] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.243798 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9334c628124870a7fe64614cb7e28d" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce" is_pre_election: true
I20260925 16:27:36.243798 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9334c628124870a7fe64614cb7e28d" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" is_pre_election: true
I20260925 16:27:36.244012 8217 raft_consensus.cc:2454] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:36.244012 8357 raft_consensus.cc:2454] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:36.244520 8011 leader_election.cc:302] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e7731dff65c34e8fa1605ebaad6bfe45, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:36.244868 8443 raft_consensus.cc:2781] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:36.244933 8443 raft_consensus.cc:482] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:36.244974 8443 raft_consensus.cc:3037] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.246428 8443 raft_consensus.cc:504] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.247134 8443 leader_election.cc:288] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Requested vote from peers e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.247602 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9334c628124870a7fe64614cb7e28d" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce"
I20260925 16:27:36.247780 8217 raft_consensus.cc:3037] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.247829 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9334c628124870a7fe64614cb7e28d" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
I20260925 16:27:36.248060 8357 raft_consensus.cc:3037] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.249020 8217 raft_consensus.cc:2454] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:36.249419 8357 raft_consensus.cc:2454] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:36.249562 8010 leader_election.cc:302] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:36.249856 8443 raft_consensus.cc:2781] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:36.249945 8443 raft_consensus.cc:686] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Running, Role: LEADER
I20260925 16:27:36.250128 8443 consensus_queue.cc:237] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.253041 7920 catalog_manager.cc:5495] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8969e5a2c4404813b1a48193955a81e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } health_report { overall_health: UNKNOWN } } }
Sep 25 16:27:36 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353654, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:36.343585 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.331996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34793 (local address 127.0.0.1:36980)
0925 16:27:36.332510 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:36.332528 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:36.332657 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:36.332992 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:36.332994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:36.333462 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:36.333598 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.333604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.334667 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.334670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.335345 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.335356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.335936 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.335940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.336019 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.336830 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:36.336845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:36.339073 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:36.341867 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:36.341883 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:36.341892 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:36.342186 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:36.342451 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:36.342455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:36.342457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:36.342533 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:36.342920 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:36.342929 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:36.343039 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 16:27:36.343139 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:27:36.343367 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":23,"thread_start_us":228,"threads_started":1}
I20260925 16:27:36.343588 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.332086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.0.98.62:34793)
0925 16:27:36.332643 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.332651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.332670 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:27:36.332770 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.332773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.332830 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.332934 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:36.333806 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.334506 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.335507 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.335775 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.335884 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.339403 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:36.339409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:36.339411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:36.339446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:36.341693 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:36.342285 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:36.342288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:36.342290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:36.342371 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:36.342606 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:36.342609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:36.342611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:36.342818 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:36.342904 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.343197 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.343429 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":472,"thread_start_us":169,"threads_started":1}
I20260925 16:27:36.349647 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36980:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 16:27:36.350800 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:36.357009 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 7891d071f48a46668e8c3347e2f617cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67e91033fff94319a31766f24c4b80de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:36.357009 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 7891d071f48a46668e8c3347e2f617cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67e91033fff94319a31766f24c4b80de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:36.357185 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 7891d071f48a46668e8c3347e2f617cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67e91033fff94319a31766f24c4b80de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 16:27:36.357525 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7891d071f48a46668e8c3347e2f617cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.357525 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7891d071f48a46668e8c3347e2f617cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.357702 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7891d071f48a46668e8c3347e2f617cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.361120 8432 tablet_bootstrap.cc:492] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: Bootstrap starting.
I20260925 16:27:36.361622 8434 tablet_bootstrap.cc:492] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: Bootstrap starting.
I20260925 16:27:36.361821 8433 tablet_bootstrap.cc:492] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: Bootstrap starting.
I20260925 16:27:36.362476 8432 tablet_bootstrap.cc:654] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.362814 8433 tablet_bootstrap.cc:654] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.362885 8434 tablet_bootstrap.cc:654] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.364112 8433 tablet_bootstrap.cc:492] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: No bootstrap required, opened a new log
I20260925 16:27:36.364112 8432 tablet_bootstrap.cc:492] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: No bootstrap required, opened a new log
I20260925 16:27:36.364226 8433 ts_tablet_manager.cc:1397] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:36.364226 8432 ts_tablet_manager.cc:1397] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:27:36.364759 8433 raft_consensus.cc:348] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.364759 8432 raft_consensus.cc:348] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.364914 8433 raft_consensus.cc:374] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.364914 8432 raft_consensus.cc:374] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.364949 8433 raft_consensus.cc:729] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.364949 8432 raft_consensus.cc:729] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7731dff65c34e8fa1605ebaad6bfe45, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.365135 8433 consensus_queue.cc:260] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.365129 8432 consensus_queue.cc:260] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.365468 8432 ts_tablet_manager.cc:1428] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:27:36.365477 8433 ts_tablet_manager.cc:1428] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:36.365491 8438 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.365826 8438 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.366115 8443 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.366292 8443 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.367250 8434 tablet_bootstrap.cc:492] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: No bootstrap required, opened a new log
I20260925 16:27:36.367365 8434 ts_tablet_manager.cc:1397] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 16:27:36.367938 8434 raft_consensus.cc:348] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.368075 8434 raft_consensus.cc:374] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.368096 8434 raft_consensus.cc:729] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d63b75f871432e94695f723a5fd2ce, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.369273 8434 consensus_queue.cc:260] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.369650 8440 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.369889 8440 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.370623 8434 ts_tablet_manager.cc:1428] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:36.624377 8440 raft_consensus.cc:482] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:36.624706 8440 raft_consensus.cc:504] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.626540 8440 leader_election.cc:288] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.631932 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.627679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60731 (local address 127.0.98.3:40383)
0925 16:27:36.627950 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.627955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.627972 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:27:36.628070 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.628073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.628125 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.628238 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:36.628623 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.629189 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.630755 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.631210 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.631311 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.631399 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.631632 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.631741 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260925 16:27:36.631959 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.627108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34419 (local address 127.0.98.1:44165)
0925 16:27:36.627413 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.627418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.627496 (+ 78us) server_negotiation.cc:407] Connection header received
0925 16:27:36.627723 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.627728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.627788 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.627975 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:27:36.628408 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.629311 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.630715 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.631207 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.631323 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.631400 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.631709 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.631813 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260925 16:27:36.631938 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.626976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44165 (local address 127.0.98.2:34419)
0925 16:27:36.627376 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:36.627393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:36.627566 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:36.627997 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:36.628002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:36.628043 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:36.628227 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.628235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.629480 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.629483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.630536 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.630549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.631354 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.631357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.631550 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.631573 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:27:36.631660 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 16:27:36.631748 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":68,"thread_start_us":138,"threads_started":1}
I20260925 16:27:36.631964 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.627549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40383 (local address 127.0.98.2:60731)
0925 16:27:36.627872 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:36.627887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:36.627964 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:36.628311 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:36.628315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:36.628343 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:27:36.628496 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.628503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.629340 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.629345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.630573 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.630588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.631358 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.631362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.631501 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.631523 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:27:36.631591 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:27:36.631683 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260925 16:27:36.632802 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891d071f48a46668e8c3347e2f617cc" candidate_uuid: "e8d63b75f871432e94695f723a5fd2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" is_pre_election: true
I20260925 16:27:36.632848 8065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891d071f48a46668e8c3347e2f617cc" candidate_uuid: "e8d63b75f871432e94695f723a5fd2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8969e5a2c4404813b1a48193955a81e0" is_pre_election: true
I20260925 16:27:36.633038 8357 raft_consensus.cc:2454] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d63b75f871432e94695f723a5fd2ce in term 0.
I20260925 16:27:36.633093 8065 raft_consensus.cc:2454] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d63b75f871432e94695f723a5fd2ce in term 0.
I20260925 16:27:36.633519 8151 leader_election.cc:302] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7731dff65c34e8fa1605ebaad6bfe45, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:36.633872 8440 raft_consensus.cc:2781] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:36.633975 8440 raft_consensus.cc:482] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:36.634006 8440 raft_consensus.cc:3037] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.635332 8440 raft_consensus.cc:504] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.635874 8440 leader_election.cc:288] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [CANDIDATE]: Term 1 election: Requested vote from peers 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.636677 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891d071f48a46668e8c3347e2f617cc" candidate_uuid: "e8d63b75f871432e94695f723a5fd2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
I20260925 16:27:36.636677 8065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7891d071f48a46668e8c3347e2f617cc" candidate_uuid: "e8d63b75f871432e94695f723a5fd2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8969e5a2c4404813b1a48193955a81e0"
I20260925 16:27:36.636878 8357 raft_consensus.cc:3037] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.636878 8065 raft_consensus.cc:3037] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.638803 8357 raft_consensus.cc:2454] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d63b75f871432e94695f723a5fd2ce in term 1.
I20260925 16:27:36.638803 8065 raft_consensus.cc:2454] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d63b75f871432e94695f723a5fd2ce in term 1.
I20260925 16:27:36.639396 8151 leader_election.cc:302] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7731dff65c34e8fa1605ebaad6bfe45, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:36.639848 8440 raft_consensus.cc:2781] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:36.640101 8440 raft_consensus.cc:686] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [term 1 LEADER]: Becoming Leader. State: Replica: e8d63b75f871432e94695f723a5fd2ce, State: Running, Role: LEADER
I20260925 16:27:36.640465 8440 consensus_queue.cc:237] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.641973 8476 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: TxnStatusTablet state changed. Reason: New leader e8d63b75f871432e94695f723a5fd2ce. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.642185 8476 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:36.642578 8478 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 16:27:36.642786 8478 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 16:27:36.643788 7920 catalog_manager.cc:5495] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d63b75f871432e94695f723a5fd2ce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:36.660983 8443 consensus_queue.cc:1035] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:36.665947 8443 consensus_queue.cc:1035] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:36.675551 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.671380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.0.98.2:34553)
0925 16:27:36.671732 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.671740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.671754 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:27:36.671834 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.671837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.671912 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.672016 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:36.672281 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.672899 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.673798 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.674065 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.674154 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.674481 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.675093 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.675201 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.675278 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.675342 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":177,"threads_started":1}
I20260925 16:27:36.675536 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.671313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34553 (local address 127.0.0.1:40918)
0925 16:27:36.671506 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:36.671526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:36.671663 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:36.672047 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:36.672050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:36.672064 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:36.672188 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.672193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.673006 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.673010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.673662 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.673670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.674166 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.674170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.674254 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.674315 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.675185 (+ 870us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.675201 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 16:27:36.675268 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:27:36.675387 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 16:27:36.679973 8357 raft_consensus.cc:1261] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Refusing update from remote peer e8d63b75f871432e94695f723a5fd2ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:36.680333 8065 raft_consensus.cc:1261] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Refusing update from remote peer e8d63b75f871432e94695f723a5fd2ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:36.681157 8476 consensus_queue.cc:1035] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:36.681439 8440 consensus_queue.cc:1035] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:36.683961 8443 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: TxnStatusTablet state changed. Reason: New leader e8d63b75f871432e94695f723a5fd2ce. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.684114 8443 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.684736 8438 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: TxnStatusTablet state changed. Reason: New leader e8d63b75f871432e94695f723a5fd2ce. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.684890 8438 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.687424 8440 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.687597 8440 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:36.689149 8438 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.689283 8438 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e7731dff65c34e8fa1605ebaad6bfe45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.692312 8476 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.692529 8476 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P e8d63b75f871432e94695f723a5fd2ce: This TxnStatusTablet replica's current role is: LEADER
I20260925 16:27:36.695104 8443 tablet_replica.cc:439] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8d63b75f871432e94695f723a5fd2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } } }
I20260925 16:27:36.695328 8443 tablet_replica.cc:442] T 7891d071f48a46668e8c3347e2f617cc P 8969e5a2c4404813b1a48193955a81e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 16:27:36.713145 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.708237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59409 (local address 127.0.98.2:34553)
0925 16:27:36.708519 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.708524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.708753 (+ 229us) server_negotiation.cc:407] Connection header received
0925 16:27:36.708946 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.708950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.709035 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.709133 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:36.709542 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.710194 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.711679 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.711977 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.712096 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.712569 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.712710 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.712782 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.712900 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.712977 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 16:27:36.713194 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.708112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34553 (local address 127.0.98.1:59409)
0925 16:27:36.708599 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:36.708631 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:36.708809 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:36.709172 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:36.709175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:36.709186 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:36.709328 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.709334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.710331 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.710335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.711487 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.711502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.712110 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.712114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.712222 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.712301 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.712806 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.712813 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:36.712892 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 16:27:36.713003 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":154,"threads_started":1}
I20260925 16:27:36.751314 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.746404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57935 (local address 127.0.98.1:44165)
0925 16:27:36.746560 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.746565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.746886 (+ 321us) server_negotiation.cc:407] Connection header received
0925 16:27:36.747094 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.747097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.747165 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.747226 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:36.747776 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.748627 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.749929 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.750292 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.750429 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.750748 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.750859 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.750941 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.751054 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.751129 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:27:36.751363 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.746297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44165 (local address 127.0.98.2:57935)
0925 16:27:36.746788 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:36.746808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:36.746978 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:36.747503 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:36.747505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:36.747512 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 16:27:36.747635 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.747642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.748756 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.748760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.749790 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:36.749802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.750435 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.750438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:36.750507 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.750594 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.750967 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.750974 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:36.751033 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 16:27:36.751138 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":194,"threads_started":1}
I20260925 16:27:36.783871 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:36.784174 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:36.787850 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Sending DeleteTablet for 3 replicas of tablet 5c9334c628124870a7fe64614cb7e28d
I20260925 16:27:36.789016 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9334c628124870a7fe64614cb7e28d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260925 16:27:36.789708 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9334c628124870a7fe64614cb7e28d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41244
I20260925 16:27:36.790094 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9334c628124870a7fe64614cb7e28d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50732
I20260925 16:27:36.790835 8502 tablet_replica.cc:330] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: stopping tablet replica
I20260925 16:27:36.791129 8502 raft_consensus.cc:2229] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:36.791388 8502 raft_consensus.cc:2258] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:36.790900 8503 tablet_replica.cc:330] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: stopping tablet replica
I20260925 16:27:36.791615 8503 raft_consensus.cc:2229] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:36.791731 8503 raft_consensus.cc:2258] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:36.792548 8502 ts_tablet_manager.cc:1905] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:36.793113 8501 tablet_replica.cc:330] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: stopping tablet replica
I20260925 16:27:36.793252 8501 raft_consensus.cc:2229] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:36.793381 8501 raft_consensus.cc:2258] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:36.796722 8503 ts_tablet_manager.cc:1905] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:36.797323 8502 ts_tablet_manager.cc:1918] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:36.797451 8502 log.cc:1196] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/5c9334c628124870a7fe64614cb7e28d
I20260925 16:27:36.797821 8502 ts_tablet_manager.cc:1939] T 5c9334c628124870a7fe64614cb7e28d P 8969e5a2c4404813b1a48193955a81e0: Deleting consensus metadata
I20260925 16:27:36.797806 8501 ts_tablet_manager.cc:1905] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:36.798842 7907 catalog_manager.cc:4841] TS 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165): tablet 5c9334c628124870a7fe64614cb7e28d (table test-table [id=3b4bb3b776ad4d069fd4555380bb36bb]) successfully deleted
I20260925 16:27:36.803048 8503 ts_tablet_manager.cc:1918] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:36.803315 8503 log.cc:1196] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/5c9334c628124870a7fe64614cb7e28d
I20260925 16:27:36.803759 8503 ts_tablet_manager.cc:1939] T 5c9334c628124870a7fe64614cb7e28d P e8d63b75f871432e94695f723a5fd2ce: Deleting consensus metadata
I20260925 16:27:36.804232 8501 ts_tablet_manager.cc:1918] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:36.804359 8501 log.cc:1196] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/5c9334c628124870a7fe64614cb7e28d
I20260925 16:27:36.804651 8501 ts_tablet_manager.cc:1939] T 5c9334c628124870a7fe64614cb7e28d P e7731dff65c34e8fa1605ebaad6bfe45: Deleting consensus metadata
I20260925 16:27:36.805265 7906 catalog_manager.cc:4841] TS e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553): tablet 5c9334c628124870a7fe64614cb7e28d (table test-table [id=3b4bb3b776ad4d069fd4555380bb36bb]) successfully deleted
I20260925 16:27:36.805744 7907 catalog_manager.cc:4841] TS e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383): tablet 5c9334c628124870a7fe64614cb7e28d (table test-table [id=3b4bb3b776ad4d069fd4555380bb36bb]) successfully deleted
Sep 25 16:27:36 dist-test-slave-31kt krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 16:27:36 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:36 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:36.822666 8467 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:36.813615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.0.98.62:34793)
0925 16:27:36.813735 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.813740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.814376 (+ 636us) server_negotiation.cc:407] Connection header received
0925 16:27:36.814516 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.814519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.814570 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.814641 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:36.815389 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.816196 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.817538 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.817874 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.817979 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.822521 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38204: BlockingRecv error: recv got EOF from 127.0.0.1:38204 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 16:27:36 dist-test-slave-31kt krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 16:27:36.839008 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.824839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.98.62:34793)
0925 16:27:36.824988 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.824993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.825376 (+ 383us) server_negotiation.cc:407] Connection header received
0925 16:27:36.825492 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.825497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.825595 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.825747 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:36.826594 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.827567 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.828619 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.828985 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.829164 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.833123 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:36.833131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:36.833135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:36.833208 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:36.835914 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:36.836880 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:36.836884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:36.836887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:36.837024 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:36.837356 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:36.837359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:36.837361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:36.837654 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:36.837843 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.838385 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.838775 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 16:27:36.842808 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:27:36.843515 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:36.851063 8057 tablet_service.cc:1467] Processing CreateTablet for tablet f624df415a074d7fa710bce289a15eb9 (DEFAULT_TABLE table=test-table [id=3127bedfd9af48a9a93f6b1c91250451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:36.851544 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f624df415a074d7fa710bce289a15eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.851594 8197 tablet_service.cc:1467] Processing CreateTablet for tablet f624df415a074d7fa710bce289a15eb9 (DEFAULT_TABLE table=test-table [id=3127bedfd9af48a9a93f6b1c91250451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:36.851527 8337 tablet_service.cc:1467] Processing CreateTablet for tablet f624df415a074d7fa710bce289a15eb9 (DEFAULT_TABLE table=test-table [id=3127bedfd9af48a9a93f6b1c91250451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:36.851972 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f624df415a074d7fa710bce289a15eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.851972 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f624df415a074d7fa710bce289a15eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.854477 8433 tablet_bootstrap.cc:492] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Bootstrap starting.
I20260925 16:27:36.855417 8433 tablet_bootstrap.cc:654] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.855684 8432 tablet_bootstrap.cc:492] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Bootstrap starting.
I20260925 16:27:36.856400 8434 tablet_bootstrap.cc:492] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Bootstrap starting.
I20260925 16:27:36.856880 8432 tablet_bootstrap.cc:654] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.857059 8433 tablet_bootstrap.cc:492] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: No bootstrap required, opened a new log
I20260925 16:27:36.857177 8433 ts_tablet_manager.cc:1397] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:36.857398 8434 tablet_bootstrap.cc:654] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.857699 8433 raft_consensus.cc:348] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.857897 8433 raft_consensus.cc:374] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.857952 8433 raft_consensus.cc:729] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.858165 8433 consensus_queue.cc:260] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.858741 8432 tablet_bootstrap.cc:492] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: No bootstrap required, opened a new log
I20260925 16:27:36.858822 8434 tablet_bootstrap.cc:492] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: No bootstrap required, opened a new log
I20260925 16:27:36.858872 8432 ts_tablet_manager.cc:1397] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 16:27:36.858927 8434 ts_tablet_manager.cc:1397] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:27:36.859481 8434 raft_consensus.cc:348] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.859630 8434 raft_consensus.cc:374] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.859663 8434 raft_consensus.cc:729] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d63b75f871432e94695f723a5fd2ce, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.859822 8434 consensus_queue.cc:260] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.860150 8434 ts_tablet_manager.cc:1428] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 16:27:36.860093 8432 raft_consensus.cc:348] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.860239 8432 raft_consensus.cc:374] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.860272 8432 raft_consensus.cc:729] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7731dff65c34e8fa1605ebaad6bfe45, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.860412 8432 consensus_queue.cc:260] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.861369 8432 ts_tablet_manager.cc:1428] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:27:36.861878 8433 ts_tablet_manager.cc:1428] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 16:27:36.869619 8496 raft_consensus.cc:482] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:36.869875 8496 raft_consensus.cc:504] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.870409 8496 leader_election.cc:288] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.870994 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f624df415a074d7fa710bce289a15eb9" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce" is_pre_election: true
I20260925 16:27:36.871065 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f624df415a074d7fa710bce289a15eb9" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" is_pre_election: true
I20260925 16:27:36.871192 8217 raft_consensus.cc:2454] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:36.871218 8357 raft_consensus.cc:2454] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:36.871621 8011 leader_election.cc:302] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e7731dff65c34e8fa1605ebaad6bfe45; no voters:
I20260925 16:27:36.871922 8496 raft_consensus.cc:2781] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:36.872009 8496 raft_consensus.cc:482] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:36.872048 8496 raft_consensus.cc:3037] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.873385 8496 raft_consensus.cc:504] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.873871 8496 leader_election.cc:288] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Requested vote from peers e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.874457 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f624df415a074d7fa710bce289a15eb9" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
I20260925 16:27:36.874457 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f624df415a074d7fa710bce289a15eb9" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce"
I20260925 16:27:36.874716 8217 raft_consensus.cc:3037] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.874716 8357 raft_consensus.cc:3037] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.876304 8217 raft_consensus.cc:2454] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:36.876304 8357 raft_consensus.cc:2454] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:36.876783 8011 leader_election.cc:302] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e7731dff65c34e8fa1605ebaad6bfe45; no voters:
I20260925 16:27:36.877107 8496 raft_consensus.cc:2781] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:36.877316 8496 raft_consensus.cc:686] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Running, Role: LEADER
I20260925 16:27:36.877494 8496 consensus_queue.cc:237] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.879853 7918 catalog_manager.cc:5495] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8969e5a2c4404813b1a48193955a81e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:36.892731 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:36.888658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.0.98.1:44165)
0925 16:27:36.888806 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:36.888810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:36.888824 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:27:36.888898 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:36.888900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:36.888956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:36.889049 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:27:36.889351 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.890019 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.891226 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:36.891510 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:36.891612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:36.892113 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:27:36.892222 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:27:36.892297 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:36.892511 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 16:27:36.892579 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:27:36.895330 8217 raft_consensus.cc:1261] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Refusing update from remote peer 8969e5a2c4404813b1a48193955a81e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:36.895871 8496 consensus_queue.cc:1035] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:36.899573 8357 raft_consensus.cc:1261] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Refusing update from remote peer 8969e5a2c4404813b1a48193955a81e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:36.900169 8439 consensus_queue.cc:1035] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:36.904539 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:36.904754 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:36.907217 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Sending DeleteTablet for 3 replicas of tablet f624df415a074d7fa710bce289a15eb9
I20260925 16:27:36.907976 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet f624df415a074d7fa710bce289a15eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50732
I20260925 16:27:36.907948 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet f624df415a074d7fa710bce289a15eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260925 16:27:36.908182 8501 tablet_replica.cc:330] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: stopping tablet replica
I20260925 16:27:36.908200 8503 tablet_replica.cc:330] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: stopping tablet replica
I20260925 16:27:36.908334 8503 raft_consensus.cc:2229] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:36.908331 8501 raft_consensus.cc:2229] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:36.908505 8501 raft_consensus.cc:2258] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:36.908509 8503 raft_consensus.cc:2258] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:36.909013 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet f624df415a074d7fa710bce289a15eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41244
I20260925 16:27:36.909276 8502 tablet_replica.cc:330] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: stopping tablet replica
I20260925 16:27:36.909405 8502 raft_consensus.cc:2229] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:36.909646 8502 raft_consensus.cc:2258] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:36.910259 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 16:27:36.910544 8503 ts_tablet_manager.cc:1905] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:27:36.910899 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:27:36.914729 8503 ts_tablet_manager.cc:1918] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:36.914863 8503 log.cc:1196] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/f624df415a074d7fa710bce289a15eb9
W20260925 16:27:36.914765 8522 consensus_queue.cc:1159] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" status { last_received { term: 1 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 2 } }
I20260925 16:27:36.915148 8503 ts_tablet_manager.cc:1939] T f624df415a074d7fa710bce289a15eb9 P e8d63b75f871432e94695f723a5fd2ce: Deleting consensus metadata
I20260925 16:27:36.916105 7906 catalog_manager.cc:4841] TS e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553): tablet f624df415a074d7fa710bce289a15eb9 (table test-table [id=3127bedfd9af48a9a93f6b1c91250451]) successfully deleted
I20260925 16:27:36.916117 8501 ts_tablet_manager.cc:1905] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:36.917017 8502 ts_tablet_manager.cc:1905] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:36.917927 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 8429b6f853de4023a04bc73311ea45e2 (DEFAULT_TABLE table=test-table [id=088ce03b352744b882159d04afd253e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:36.918098 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 8429b6f853de4023a04bc73311ea45e2 (DEFAULT_TABLE table=test-table [id=088ce03b352744b882159d04afd253e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:36.918211 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8429b6f853de4023a04bc73311ea45e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.918387 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8429b6f853de4023a04bc73311ea45e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.918957 8501 ts_tablet_manager.cc:1918] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:36.919031 8501 log.cc:1196] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/f624df415a074d7fa710bce289a15eb9
I20260925 16:27:36.919270 8501 ts_tablet_manager.cc:1939] T f624df415a074d7fa710bce289a15eb9 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting consensus metadata
I20260925 16:27:36.920225 7907 catalog_manager.cc:4841] TS e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383): tablet f624df415a074d7fa710bce289a15eb9 (table test-table [id=3127bedfd9af48a9a93f6b1c91250451]) successfully deleted
I20260925 16:27:36.921020 8433 tablet_bootstrap.cc:492] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0: Bootstrap starting.
I20260925 16:27:36.921806 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 8429b6f853de4023a04bc73311ea45e2 (DEFAULT_TABLE table=test-table [id=088ce03b352744b882159d04afd253e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:27:36.922175 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8429b6f853de4023a04bc73311ea45e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:36.922232 8432 tablet_bootstrap.cc:492] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Bootstrap starting.
I20260925 16:27:36.923553 8432 tablet_bootstrap.cc:654] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.923707 8502 ts_tablet_manager.cc:1918] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:27:36.923797 8502 log.cc:1196] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/f624df415a074d7fa710bce289a15eb9
I20260925 16:27:36.924058 8502 ts_tablet_manager.cc:1939] T f624df415a074d7fa710bce289a15eb9 P 8969e5a2c4404813b1a48193955a81e0: Deleting consensus metadata
I20260925 16:27:36.925247 8434 tablet_bootstrap.cc:492] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Bootstrap starting.
I20260925 16:27:36.925985 8433 tablet_bootstrap.cc:654] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.926167 8434 tablet_bootstrap.cc:654] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:36.926854 7907 catalog_manager.cc:4841] TS 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165): tablet f624df415a074d7fa710bce289a15eb9 (table test-table [id=3127bedfd9af48a9a93f6b1c91250451]) successfully deleted
I20260925 16:27:36.927497 8432 tablet_bootstrap.cc:492] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: No bootstrap required, opened a new log
I20260925 16:27:36.927619 8432 ts_tablet_manager.cc:1397] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 16:27:36.927636 8433 tablet_bootstrap.cc:492] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0: No bootstrap required, opened a new log
I20260925 16:27:36.927736 8433 ts_tablet_manager.cc:1397] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 16:27:36.927901 8434 tablet_bootstrap.cc:492] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: No bootstrap required, opened a new log
I20260925 16:27:36.927994 8434 ts_tablet_manager.cc:1397] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 16:27:36.928191 8432 raft_consensus.cc:348] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.928337 8432 raft_consensus.cc:374] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.928272 8433 raft_consensus.cc:348] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.928375 8432 raft_consensus.cc:729] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7731dff65c34e8fa1605ebaad6bfe45, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.928409 8433 raft_consensus.cc:374] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.928447 8433 raft_consensus.cc:729] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.928438 8434 raft_consensus.cc:348] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.928557 8434 raft_consensus.cc:374] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:36.928511 8432 consensus_queue.cc:260] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.928587 8434 raft_consensus.cc:729] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d63b75f871432e94695f723a5fd2ce, State: Initialized, Role: FOLLOWER
I20260925 16:27:36.928646 8433 consensus_queue.cc:260] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.928709 8434 consensus_queue.cc:260] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.928815 8432 ts_tablet_manager.cc:1428] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:36.928957 8433 ts_tablet_manager.cc:1428] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 16:27:36.928972 8434 ts_tablet_manager.cc:1428] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 16:27:36.977110 8496 raft_consensus.cc:482] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:27:36.977356 8496 raft_consensus.cc:504] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.978016 8496 leader_election.cc:288] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.978650 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429b6f853de4023a04bc73311ea45e2" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" is_pre_election: true
I20260925 16:27:36.978663 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429b6f853de4023a04bc73311ea45e2" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce" is_pre_election: true
I20260925 16:27:36.978860 8357 raft_consensus.cc:2454] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:36.978897 8217 raft_consensus.cc:2454] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 0.
I20260925 16:27:36.979275 8011 leader_election.cc:302] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e7731dff65c34e8fa1605ebaad6bfe45; no voters:
I20260925 16:27:36.979497 8496 raft_consensus.cc:2781] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:27:36.979575 8496 raft_consensus.cc:482] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:27:36.979620 8496 raft_consensus.cc:3037] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.981125 8496 raft_consensus.cc:504] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.981587 8496 leader_election.cc:288] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Requested vote from peers e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553), e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383)
I20260925 16:27:36.982136 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429b6f853de4023a04bc73311ea45e2" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d63b75f871432e94695f723a5fd2ce"
I20260925 16:27:36.982203 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429b6f853de4023a04bc73311ea45e2" candidate_uuid: "8969e5a2c4404813b1a48193955a81e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7731dff65c34e8fa1605ebaad6bfe45"
I20260925 16:27:36.982335 8217 raft_consensus.cc:3037] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.982439 8357 raft_consensus.cc:3037] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:36.983489 8217 raft_consensus.cc:2454] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:36.983596 8357 raft_consensus.cc:2454] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8969e5a2c4404813b1a48193955a81e0 in term 1.
I20260925 16:27:36.983949 8010 leader_election.cc:302] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8969e5a2c4404813b1a48193955a81e0, e7731dff65c34e8fa1605ebaad6bfe45, e8d63b75f871432e94695f723a5fd2ce; no voters:
I20260925 16:27:36.984195 8496 raft_consensus.cc:2781] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:36.984303 8496 raft_consensus.cc:686] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8969e5a2c4404813b1a48193955a81e0, State: Running, Role: LEADER
I20260925 16:27:36.984468 8496 consensus_queue.cc:237] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } }
I20260925 16:27:36.986500 7920 catalog_manager.cc:5495] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8969e5a2c4404813b1a48193955a81e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8969e5a2c4404813b1a48193955a81e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 } health_report { overall_health: UNKNOWN } } }
I20260925 16:27:37.016064 8217 raft_consensus.cc:1261] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Refusing update from remote peer 8969e5a2c4404813b1a48193955a81e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:37.016126 8357 raft_consensus.cc:1261] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Refusing update from remote peer 8969e5a2c4404813b1a48193955a81e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:27:37.016528 8496 consensus_queue.cc:1035] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d63b75f871432e94695f723a5fd2ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:37.016784 8439 consensus_queue.cc:1035] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7731dff65c34e8fa1605ebaad6bfe45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:27:37.059676 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:37.059917 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:27:37.062407 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3e5e49ea5594f0388aa2da8d89f3289: Sending DeleteTablet for 3 replicas of tablet 8429b6f853de4023a04bc73311ea45e2
I20260925 16:27:37.063215 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429b6f853de4023a04bc73311ea45e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50732
I20260925 16:27:37.063306 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429b6f853de4023a04bc73311ea45e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41244
I20260925 16:27:37.063315 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429b6f853de4023a04bc73311ea45e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:27:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260925 16:27:37.063517 8503 tablet_replica.cc:330] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: stopping tablet replica
I20260925 16:27:37.063692 8503 raft_consensus.cc:2229] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:37.063772 8501 tablet_replica.cc:330] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: stopping tablet replica
I20260925 16:27:37.063860 8503 raft_consensus.cc:2258] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:37.064087 8502 tablet_replica.cc:330] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0: stopping tablet replica
I20260925 16:27:37.064098 8501 raft_consensus.cc:2229] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:27:37.064235 8502 raft_consensus.cc:2229] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:27:37.064374 8501 raft_consensus.cc:2258] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:37.064510 8502 raft_consensus.cc:2258] T 8429b6f853de4023a04bc73311ea45e2 P 8969e5a2c4404813b1a48193955a81e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:27:37.064653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 7992
I20260925 16:27:37.066637 8503 ts_tablet_manager.cc:1905] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:37.067301 8501 ts_tablet_manager.cc:1905] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:27:37.070230 8503 ts_tablet_manager.cc:1918] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:37.070375 8503 log.cc:1196] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/8429b6f853de4023a04bc73311ea45e2
I20260925 16:27:37.070463 8501 ts_tablet_manager.cc:1918] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 16:27:37.070552 8501 log.cc:1196] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/8429b6f853de4023a04bc73311ea45e2
I20260925 16:27:37.070641 8503 ts_tablet_manager.cc:1939] T 8429b6f853de4023a04bc73311ea45e2 P e8d63b75f871432e94695f723a5fd2ce: Deleting consensus metadata
I20260925 16:27:37.070823 8501 ts_tablet_manager.cc:1939] T 8429b6f853de4023a04bc73311ea45e2 P e7731dff65c34e8fa1605ebaad6bfe45: Deleting consensus metadata
I20260925 16:27:37.071856 7907 catalog_manager.cc:4841] TS e7731dff65c34e8fa1605ebaad6bfe45 (127.0.98.3:40383): tablet 8429b6f853de4023a04bc73311ea45e2 (table test-table [id=088ce03b352744b882159d04afd253e7]) successfully deleted
I20260925 16:27:37.071910 7906 catalog_manager.cc:4841] TS e8d63b75f871432e94695f723a5fd2ce (127.0.98.2:34553): tablet 8429b6f853de4023a04bc73311ea45e2 (table test-table [id=088ce03b352744b882159d04afd253e7]) successfully deleted
W20260925 16:27:37.073338 7907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44165 (error 108)
I20260925 16:27:37.073818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 8132
I20260925 16:27:37.076720 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:37.076032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44165 (local address 127.0.0.1:37922)
0925 16:27:37.076478 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:37.076594 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
W20260925 16:27:37.076925 7907 catalog_manager.cc:4584] TS 8969e5a2c4404813b1a48193955a81e0 (127.0.98.1:44165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8429b6f853de4023a04bc73311ea45e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44165: connect: Connection refused (error 111)
I20260925 16:27:37.081907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 8272
I20260925 16:27:37.087987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 7888
2026-09-25T16:27:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3985 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:37 dist-test-slave-31kt krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:37 dist-test-slave-31kt krb5kdc[8533](info): set up 2 sockets
Sep 25 16:27:37 dist-test-slave-31kt krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260925 16:27:38.083652 392 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:38 dist-test-slave-31kt krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:38.252117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:38.402484 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:38.402931 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:38.402993 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:38.408571 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:38.408679 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:38.408701 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:38.408725 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:38.408742 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:38.415112 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:38.416877 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:38.418584 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:38.431135 8552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:38.431170 8554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:38.431389 8546 server_base.cc:1034] running on GCE node
W20260925 16:27:38.431170 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:38.432456 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:38.434748 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:38.438277 8546 hybrid_clock.cc:648] HybridClock initialized: now 1790353658438195 us; error 104 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T16:27:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1365 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:38 dist-test-slave-31kt krb5kdc[8559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:38 dist-test-slave-31kt krb5kdc[8559](info): set up 2 sockets
Sep 25 16:27:38 dist-test-slave-31kt krb5kdc[8559](info): commencing operation
krb5kdc: starting...
W20260925 16:27:39.483692 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:27:39 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:27:39.625671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38213
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:39.748327 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:39.748664 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:39.748709 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:39.753063 8572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:39.753161 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:39.753180 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:39.753197 8572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:39.753211 8572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:39.758970 8572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:39.760183 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:39.761288 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:39.771555 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:39.771555 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:39.771608 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:39.772037 8572 server_base.cc:1034] running on GCE node
I20260925 16:27:39.772862 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:39.774683 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:39.775971 8572 hybrid_clock.cc:648] HybridClock initialized: now 1790353659775934 us; error 69 us; skew 500 ppm
Sep 25 16:27:39 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:39.785140 8572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:39.786598 8572 webserver.cc:460] Webserver started at http://127.0.98.62:43499/ using document root <none> and password file <none>
I20260925 16:27:39.787372 8572 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:39.787451 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:39.787700 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:39.790360 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "93f4c0399662470094d4ecf5f046c851"
format_stamp: "Formatted at 2026-09-25 16:27:39 on dist-test-slave-31kt"
server_key: "b59bccb2f21898d030d46cae4b107edc"
server_key_iv: "bd7c9651619c06910333295a2c08b6de"
server_key_version: "encryptionkey@0"
I20260925 16:27:39.791007 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "93f4c0399662470094d4ecf5f046c851"
format_stamp: "Formatted at 2026-09-25 16:27:39 on dist-test-slave-31kt"
server_key: "b59bccb2f21898d030d46cae4b107edc"
server_key_iv: "bd7c9651619c06910333295a2c08b6de"
server_key_version: "encryptionkey@0"
I20260925 16:27:39.796407 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 16:27:39.800065 8587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:39.802093 8572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:39.802311 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "93f4c0399662470094d4ecf5f046c851"
format_stamp: "Formatted at 2026-09-25 16:27:39 on dist-test-slave-31kt"
server_key: "b59bccb2f21898d030d46cae4b107edc"
server_key_iv: "bd7c9651619c06910333295a2c08b6de"
server_key_version: "encryptionkey@0"
I20260925 16:27:39.802474 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:39.835850 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:39.840337 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:39.840584 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:39.850135 8572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38213
I20260925 16:27:39.850303 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38213 every 8 connection(s)
I20260925 16:27:39.851521 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:39.855026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 8572
I20260925 16:27:39.855190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:39.855552 392 external_mini_cluster.cc:1428] Setting key 9fb1e698d832b2fa1afe4684613a54f6
I20260925 16:27:39.856063 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:39.864616 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Bootstrap starting.
Sep 25 16:27:39 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:39.867780 8640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:39.868724 8640 log.cc:824] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:39.871883 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: No bootstrap required, opened a new log
I20260925 16:27:39.875309 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:39.857497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.98.62:38213)
0925 16:27:39.858253 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:39.858274 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:39.858328 (+ 54us) server_negotiation.cc:407] Connection header received
0925 16:27:39.859180 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:39.859206 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:39.859724 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:39.860182 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:39.861249 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:39.862774 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:39.864027 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:39.864772 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:39.864985 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:39.868454 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:39.868463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:39.868485 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:39.868534 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:39.871020 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:39.871720 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:39.871727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:39.871734 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:39.871935 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:39.872358 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:39.872361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:39.872363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:39.872982 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:39.873201 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:39.873578 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 16:27:39.874011 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":281,"threads_started":1}
I20260925 16:27:39.877064 8640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:39.877358 8640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:39.877409 8640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f4c0399662470094d4ecf5f046c851, State: Initialized, Role: FOLLOWER
I20260925 16:27:39.877985 8640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:39.878126 8640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:39.878194 8640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:39.878302 8640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:39.879421 8640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:39.879812 8640 leader_election.cc:302] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f4c0399662470094d4ecf5f046c851; no voters:
I20260925 16:27:39.880169 8640 leader_election.cc:288] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:39.880384 8645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:39.880759 8645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 LEADER]: Becoming Leader. State: Replica: 93f4c0399662470094d4ecf5f046c851, State: Running, Role: LEADER
I20260925 16:27:39.881230 8645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:39.881660 8640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:39.883587 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f4c0399662470094d4ecf5f046c851. Latest consensus state: current_term: 1 leader_uuid: "93f4c0399662470094d4ecf5f046c851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } } }
I20260925 16:27:39.883808 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:39.884128 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f4c0399662470094d4ecf5f046c851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } } }
I20260925 16:27:39.884378 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: This master's current role is: LEADER
W20260925 16:27:39.888940 8660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:27:39.889074 8660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:27:39.889211 8656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:39.890376 8656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:39.898931 8656 catalog_manager.cc:1347] Generated new cluster ID: 8e58e6bd2fee4d1b981c499d1c17c45c
I20260925 16:27:39.899143 8656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:39.922567 8656 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:39.924163 8656 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:39.931674 8656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Generated new TSK 0
I20260925 16:27:39.932781 8656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:27:39.970530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38213
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:40.121809 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:40.122500 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:40.122572 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:40.127957 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:40.128064 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:40.128135 8666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:40.134930 8666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:40.136757 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:40.138368 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:40.151645 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:40.151895 8666 server_base.cc:1034] running on GCE node
W20260925 16:27:40.151651 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:40.152108 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:40.152624 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:40.153434 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:40.154654 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790353660154613 us; error 82 us; skew 500 ppm
Sep 25 16:27:40 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:40.158175 8666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:27:40.159369 8666 webserver.cc:460] Webserver started at http://127.0.98.1:38713/ using document root <none> and password file <none>
I20260925 16:27:40.160039 8666 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:40.160123 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:40.160337 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:40.162679 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "693d1f8f9dae4ff1b2b102dfff178863"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "1855b8ced42222b27db311138fc4fa01"
server_key_iv: "e66570b70f419a15c78015d4bfb350f2"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.163304 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "693d1f8f9dae4ff1b2b102dfff178863"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "1855b8ced42222b27db311138fc4fa01"
server_key_iv: "e66570b70f419a15c78015d4bfb350f2"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.168226 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:27:40.172607 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.174739 8666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:40.174955 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "693d1f8f9dae4ff1b2b102dfff178863"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "1855b8ced42222b27db311138fc4fa01"
server_key_iv: "e66570b70f419a15c78015d4bfb350f2"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.175134 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:40.194001 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:40.198688 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:40.198957 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:40.199800 8666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:40.201046 8666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:40.201115 8666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.201182 8666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:40.201211 8666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.212905 8666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41217
I20260925 16:27:40.213061 8794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41217 every 8 connection(s)
I20260925 16:27:40.214229 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:40.221393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 8666
I20260925 16:27:40.221546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:40.221908 392 external_mini_cluster.cc:1428] Setting key 327f92e4fe08089857993b39a5eed02b
Sep 25 16:27:40 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:40.234740 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:40.217217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57449 (local address 127.0.98.62:38213)
0925 16:27:40.217458 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:40.217464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:40.219163 (+ 1699us) server_negotiation.cc:407] Connection header received
0925 16:27:40.220892 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:40.220898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:40.220997 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:40.221175 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:40.223494 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.224258 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.226107 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.226438 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.226511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:40.229978 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:40.229983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:40.229986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:40.230020 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:40.231841 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:40.232645 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:40.232649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:40.232651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:40.232748 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:40.233176 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:40.233179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:40.233181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:40.233428 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:40.233512 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:40.234338 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 16:27:40.234520 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 16:27:40.236778 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:40.217928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:57449)
0925 16:27:40.218860 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:40.218936 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:40.220532 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:40.221558 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:40.221598 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:40.222469 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:40.223287 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:40.223311 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.224449 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.224453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:40.225934 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:40.225952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.226753 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.226757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:40.226990 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:40.227712 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:40.227734 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:40.229668 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:40.232050 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:40.232058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:40.232082 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:40.232503 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:40.232854 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:40.232857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:40.232859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:40.233068 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:40.233584 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:40.233591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:40.234017 (+ 426us) client_negotiation.cc:769] Sending connection context
0925 16:27:40.235044 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0925 16:27:40.235503 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":244,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:27:40.239355 8795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38213
I20260925 16:27:40.239771 8795 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:40.240693 8795 heartbeater.cc:511] Master 127.0.98.62:38213 requested a full tablet report, sending...
I20260925 16:27:40.245476 8604 ts_manager.cc:194] Registered new tserver with Master: 693d1f8f9dae4ff1b2b102dfff178863 (127.0.98.1:41217)
I20260925 16:27:40.247254 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:27:40.271019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38213
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:40.403837 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:40.404177 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:40.404223 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:40.408622 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:40.408699 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:40.408763 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:40.414552 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:40.415738 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:40.416879 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:40.428216 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:40.428231 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:40.428231 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:40.428643 8800 server_base.cc:1034] running on GCE node
I20260925 16:27:40.429401 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:40.430245 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:40.431497 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790353660431471 us; error 47 us; skew 500 ppm
Sep 25 16:27:40 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:40.434998 8800 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:27:40.436147 8800 webserver.cc:460] Webserver started at http://127.0.98.2:45053/ using document root <none> and password file <none>
I20260925 16:27:40.436779 8800 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:40.436860 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:40.437041 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:40.439539 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "a4b5b0204fbe433892a14bd623f223c1"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "5bc11dd8ce13802369927cd0a8df9a34"
server_key_iv: "35262cf3a7e4406208c1f49470677b77"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.440680 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a4b5b0204fbe433892a14bd623f223c1"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "5bc11dd8ce13802369927cd0a8df9a34"
server_key_iv: "35262cf3a7e4406208c1f49470677b77"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.447713 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260925 16:27:40.451753 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.453894 8800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:27:40.454075 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "a4b5b0204fbe433892a14bd623f223c1"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "5bc11dd8ce13802369927cd0a8df9a34"
server_key_iv: "35262cf3a7e4406208c1f49470677b77"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.454344 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:40.470386 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:40.474031 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:40.474225 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:40.474875 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:40.475926 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:40.475994 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.476039 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:40.476053 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.488683 8800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46429
I20260925 16:27:40.488794 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46429 every 8 connection(s)
I20260925 16:27:40.489809 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:40.490904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 8800
I20260925 16:27:40.491063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:40.491452 392 external_mini_cluster.cc:1428] Setting key 71eb37f2e439aa0943b856fa82f5b01e
Sep 25 16:27:40 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:27:40.512791 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:40.493904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49955 (local address 127.0.98.62:38213)
0925 16:27:40.494046 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:40.494051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:40.494670 (+ 619us) server_negotiation.cc:407] Connection header received
0925 16:27:40.497009 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:40.497014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:40.497119 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:40.497291 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:40.499245 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.499930 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.502281 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.502558 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.502676 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:40.507006 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:40.507010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:40.507013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:40.507046 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:40.509036 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:40.510055 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:40.510062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:40.510063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:40.510234 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:40.510861 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:40.510866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:40.510868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:40.511251 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:40.511400 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:40.512331 (+ 931us) server_negotiation.cc:299] Negotiation successful
0925 16:27:40.512556 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:27:40.514096 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:40.493268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:49955)
0925 16:27:40.494227 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:40.494381 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:40.496578 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:40.497583 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:40.497594 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:40.498197 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:40.498948 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:40.498969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.500155 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.500159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:40.501974 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:40.501997 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.502805 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.502811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:40.503129 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:40.504170 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:40.504203 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:40.506720 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:40.509306 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:40.509315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:40.509336 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:40.509811 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:40.510376 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:40.510380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:40.510382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:40.510636 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:40.511471 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:40.511482 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:40.511964 (+ 482us) client_negotiation.cc:769] Sending connection context
0925 16:27:40.512215 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 16:27:40.512672 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":197,"threads_started":1}
I20260925 16:27:40.516062 8929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38213
I20260925 16:27:40.516376 8929 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:40.517055 8929 heartbeater.cc:511] Master 127.0.98.62:38213 requested a full tablet report, sending...
I20260925 16:27:40.518584 8604 ts_manager.cc:194] Registered new tserver with Master: a4b5b0204fbe433892a14bd623f223c1 (127.0.98.2:46429)
I20260925 16:27:40.519282 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:27:40.530061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38213
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:40.664490 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:40.664938 8934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:40.664994 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:40.670662 8934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:40.670768 8934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:40.670846 8934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:40.677717 8934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:40.679330 8934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:40.680526 8934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:40.692651 8939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:40.692651 8940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:40.692922 8934 server_base.cc:1034] running on GCE node
W20260925 16:27:40.692660 8942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:40.693730 8934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:40.694650 8934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:40.695895 8934 hybrid_clock.cc:648] HybridClock initialized: now 1790353660695868 us; error 48 us; skew 500 ppm
Sep 25 16:27:40 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:40.699740 8934 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:27:40.700915 8934 webserver.cc:460] Webserver started at http://127.0.98.3:37399/ using document root <none> and password file <none>
I20260925 16:27:40.701546 8934 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:40.701623 8934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:40.701884 8934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:40.704571 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "07d3abb5c65544fc89537717e31891d3"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "277cd0135e293f42b03103bb1ceb8501"
server_key_iv: "7273c1b96c44db3542286283c428801a"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.705492 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07d3abb5c65544fc89537717e31891d3"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "277cd0135e293f42b03103bb1ceb8501"
server_key_iv: "7273c1b96c44db3542286283c428801a"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.711459 8934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 16:27:40.714766 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.717233 8934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:40.717434 8934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "07d3abb5c65544fc89537717e31891d3"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "277cd0135e293f42b03103bb1ceb8501"
server_key_iv: "7273c1b96c44db3542286283c428801a"
server_key_version: "encryptionkey@0"
I20260925 16:27:40.717582 8934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:40.739080 8934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:40.744005 8934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:40.744335 8934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:40.745179 8934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:40.747437 8934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:40.747565 8934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.747643 8934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:40.747679 8934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:40.759536 8934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41629
I20260925 16:27:40.759610 9062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41629 every 8 connection(s)
I20260925 16:27:40.760685 8934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:40.769533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 8934
I20260925 16:27:40.769702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:40.770043 392 external_mini_cluster.cc:1428] Setting key 0d56fa39740315689a1b299136c1af2b
Sep 25 16:27:40 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:27:40.780117 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:40.763714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34049 (local address 127.0.98.62:38213)
0925 16:27:40.763879 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:40.763884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:40.765040 (+ 1156us) server_negotiation.cc:407] Connection header received
0925 16:27:40.767054 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:40.767059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:40.767236 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:40.767368 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:40.769540 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.770264 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.772004 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.772255 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.772361 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:40.775532 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:40.775536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:40.775538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:40.775571 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:40.777318 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:40.778152 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:40.778156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:40.778158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:40.778263 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:40.778656 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:40.778659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:40.778660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:40.778908 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:40.779026 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:40.779702 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 16:27:40.779878 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:27:40.781459 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:40.764034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:34049)
0925 16:27:40.764767 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:40.764839 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:40.766664 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:40.767697 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:40.767708 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:40.768393 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:40.769258 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:40.769282 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.770447 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.770450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:40.771871 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:40.771885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:40.772444 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:40.772448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:40.772676 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:40.773434 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:40.773457 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:40.775275 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:40.777517 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:40.777525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:40.777548 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:40.778007 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:40.778360 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:27:40.778362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:27:40.778364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:27:40.778549 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:27:40.779038 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:40.779047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:40.779412 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 16:27:40.779631 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 16:27:40.780211 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":158,"threads_started":1}
I20260925 16:27:40.783586 9063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38213
I20260925 16:27:40.783981 9063 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:40.784855 9063 heartbeater.cc:511] Master 127.0.98.62:38213 requested a full tablet report, sending...
I20260925 16:27:40.786523 8604 ts_manager.cc:194] Registered new tserver with Master: 07d3abb5c65544fc89537717e31891d3 (127.0.98.3:41629)
I20260925 16:27:40.787263 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34049
I20260925 16:27:40.799065 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 16:27:40.889667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 8666
I20260925 16:27:40.896684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 8800
I20260925 16:27:40.902717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 8934
I20260925 16:27:40.908720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 8572
I20260925 16:27:40.917348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38213
--webserver_interface=127.0.98.62
--webserver_port=43499
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:27:41.066843 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:41.067324 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:41.067399 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:41.072602 9072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:41.072755 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:41.072789 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:41.072813 9072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:41.072834 9072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:41.080668 9072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:41.082448 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:41.084028 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:27:41.095814 9072 server_base.cc:1034] running on GCE node
W20260925 16:27:41.095734 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.095734 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.095741 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:41.097200 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:41.099186 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:41.102679 9072 hybrid_clock.cc:648] HybridClock initialized: now 1790353661102598 us; error 105 us; skew 500 ppm
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:41.106762 9072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:27:41.108345 9072 webserver.cc:460] Webserver started at http://127.0.98.62:43499/ using document root <none> and password file <none>
I20260925 16:27:41.109442 9072 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:41.109571 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:41.115252 9072 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:41.118886 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.120831 9072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:41.121042 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "93f4c0399662470094d4ecf5f046c851"
format_stamp: "Formatted at 2026-09-25 16:27:39 on dist-test-slave-31kt"
server_key: "b59bccb2f21898d030d46cae4b107edc"
server_key_iv: "bd7c9651619c06910333295a2c08b6de"
server_key_version: "encryptionkey@0"
I20260925 16:27:41.121889 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:41.143776 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:41.148469 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:41.148706 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:41.158063 9072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38213
I20260925 16:27:41.158245 9139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38213 every 8 connection(s)
I20260925 16:27:41.159586 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:41.164899 9140 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:27:41.166885 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Bootstrap starting.
I20260925 16:27:41.168520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9072
I20260925 16:27:41.169049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41217
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38213
--webserver_port=38713
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:27:41.184994 9140 log.cc:824] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:41.192602 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:27:41.193040 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Bootstrap complete.
I20260925 16:27:41.197577 9140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:41.198110 9140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f4c0399662470094d4ecf5f046c851, State: Initialized, Role: FOLLOWER
I20260925 16:27:41.198689 9140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:41.198851 9140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:41.198925 9140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:41.199025 9140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:27:41.201339 9140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:41.201748 9140 leader_election.cc:302] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f4c0399662470094d4ecf5f046c851; no voters:
I20260925 16:27:41.202169 9140 leader_election.cc:288] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:27:41.202293 9144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:27:41.202504 9144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [term 2 LEADER]: Becoming Leader. State: Replica: 93f4c0399662470094d4ecf5f046c851, State: Running, Role: LEADER
I20260925 16:27:41.202876 9144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } }
I20260925 16:27:41.203665 9140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:41.204555 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93f4c0399662470094d4ecf5f046c851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } } }
I20260925 16:27:41.204721 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:41.205030 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f4c0399662470094d4ecf5f046c851. Latest consensus state: current_term: 2 leader_uuid: "93f4c0399662470094d4ecf5f046c851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f4c0399662470094d4ecf5f046c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38213 } } }
I20260925 16:27:41.205130 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:41.206163 9153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:41.209586 9153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:41.210486 9153 catalog_manager.cc:1259] Loaded cluster ID: 8e58e6bd2fee4d1b981c499d1c17c45c
I20260925 16:27:41.210552 9153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:41.212556 9153 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:41.213231 9153 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93f4c0399662470094d4ecf5f046c851: Loaded TSK: 0
I20260925 16:27:41.214587 9153 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:27:41.324771 9142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:41.325145 9142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:41.325213 9142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:41.330643 9142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:41.330742 9142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:41.330834 9142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:41.337821 9142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38713
--tserver_master_addrs=127.0.98.62:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:41.339461 9142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:41.341382 9142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:41.355782 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.355816 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:41.356104 9142 server_base.cc:1034] running on GCE node
W20260925 16:27:41.356150 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:41.356647 9142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:41.357540 9142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:41.360939 9142 hybrid_clock.cc:648] HybridClock initialized: now 1790353661360837 us; error 125 us; skew 500 ppm
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353661, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:41.365880 9142 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 16:27:41.367388 9142 webserver.cc:460] Webserver started at http://127.0.98.1:38713/ using document root <none> and password file <none>
I20260925 16:27:41.368458 9142 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:41.368629 9142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:41.374577 9142 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:41.377161 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.378944 9142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:27:41.379092 9142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "693d1f8f9dae4ff1b2b102dfff178863"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "1855b8ced42222b27db311138fc4fa01"
server_key_iv: "e66570b70f419a15c78015d4bfb350f2"
server_key_version: "encryptionkey@0"
I20260925 16:27:41.379971 9142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:41.401712 9142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:41.406487 9142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:41.406692 9142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:41.407519 9142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:41.409029 9142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:41.409118 9142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.409271 9142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:41.409353 9142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.420490 9142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41217
I20260925 16:27:41.420641 9289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41217 every 8 connection(s)
I20260925 16:27:41.421803 9142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:41.429062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9142
I20260925 16:27:41.429656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46429
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38213
--webserver_port=45053
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:41.444917 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.424901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40069 (local address 127.0.98.62:38213)
0925 16:27:41.425596 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.425618 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.426188 (+ 570us) server_negotiation.cc:407] Connection header received
0925 16:27:41.428306 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.428321 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.428960 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.429427 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.431483 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.433634 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.436373 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.437144 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.437412 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.444014 (+ 6602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40069: BlockingRecv error: recv got EOF from 127.0.98.1:40069 (error 108)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":154,"threads_started":1}
I20260925 16:27:41.444949 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.425170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:40069)
0925 16:27:41.425927 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.425995 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.427640 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.429588 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.429599 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.430244 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.431083 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.431112 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.434015 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.434021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.435986 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.436028 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.437563 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.437642 (+ 79us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.438005 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.439043 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.439076 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.443650 (+ 4574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":196,"threads_started":1}
W20260925 16:27:41.445996 9290 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38213 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38213: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:41.454699 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.446702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:54089)
0925 16:27:41.446948 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.446971 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.447093 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.447532 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.447535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.447920 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.448178 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.448188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.449451 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.449454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.450185 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.450197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.450885 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.450890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.451000 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.451762 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.451780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.454530 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260925 16:27:41.454947 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.446862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54089 (local address 127.0.98.62:38213)
0925 16:27:41.447065 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.447071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.447100 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:27:41.447229 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.447232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.447304 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.447411 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.448341 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.449274 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.450319 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.450689 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.450823 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.454787 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54089: BlockingRecv error: recv got EOF from 127.0.98.1:54089 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:41.462764 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.455788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:37201)
0925 16:27:41.455960 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.455979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.456093 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.456423 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.456426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.456645 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.456777 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.456784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.457928 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.457931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.458791 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.458803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.459368 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.459371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.459459 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.460119 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.460137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.462596 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 16:27:41.463045 9290 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:27:41.463204 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.455916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37201 (local address 127.0.98.62:38213)
0925 16:27:41.456034 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.456037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.456051 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:27:41.456235 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.456238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.456302 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.456384 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.457047 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.457662 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.458976 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.459243 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.459350 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.462842 (+ 3492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37201: BlockingRecv error: recv got EOF from 127.0.98.1:37201 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 16:27:41.582656 9294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:41.583156 9294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:41.583237 9294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:41.589037 9294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:41.589196 9294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:41.589313 9294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:41.596489 9294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45053
--tserver_master_addrs=127.0.98.62:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:41.598263 9294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:41.599769 9294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:41.614042 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.614123 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.614055 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:41.614411 9294 server_base.cc:1034] running on GCE node
I20260925 16:27:41.615167 9294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:41.616123 9294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:41.617412 9294 hybrid_clock.cc:648] HybridClock initialized: now 1790353661617382 us; error 58 us; skew 500 ppm
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353661, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:41.622185 9294 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 16:27:41.623803 9294 webserver.cc:460] Webserver started at http://127.0.98.2:45053/ using document root <none> and password file <none>
I20260925 16:27:41.624671 9294 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:41.624804 9294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:41.630857 9294 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:27:41.634805 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.636972 9294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:41.637219 9294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "a4b5b0204fbe433892a14bd623f223c1"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "5bc11dd8ce13802369927cd0a8df9a34"
server_key_iv: "35262cf3a7e4406208c1f49470677b77"
server_key_version: "encryptionkey@0"
I20260925 16:27:41.638151 9294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:41.673097 9294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:41.678062 9294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:41.678342 9294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:41.679227 9294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:41.680493 9294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:41.680555 9294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.680626 9294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:41.680655 9294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.693513 9294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46429
I20260925 16:27:41.693563 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46429 every 8 connection(s)
I20260925 16:27:41.695071 9294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:41.702335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9294
I20260925 16:27:41.702893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41629
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38213
--webserver_port=37399
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:41.711174 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.698023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53067 (local address 127.0.98.62:38213)
0925 16:27:41.698230 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.698233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.699761 (+ 1528us) server_negotiation.cc:407] Connection header received
0925 16:27:41.701416 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.701419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.701502 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.701581 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.703219 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.703811 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.705266 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.705539 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.705616 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.710995 (+ 5379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53067: BlockingRecv error: recv got EOF from 127.0.98.2:53067 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 16:27:41.712430 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.698645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:53067)
0925 16:27:41.699502 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.699588 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.701120 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.701895 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.701905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.702360 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.702987 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.703008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.703951 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.703954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.705109 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.705124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.705874 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.705877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.706136 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.707084 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.707113 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.710735 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":194,"threads_started":1}
W20260925 16:27:41.713313 9423 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38213 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38213: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:41.721604 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.714130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:46923)
0925 16:27:41.714311 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.714333 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.714503 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.714929 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.714933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.715192 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.715381 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.715391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.716280 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.716284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.717300 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.717313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.718075 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.718080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.718205 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.718908 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.718923 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.721408 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 16:27:41.721920 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.714299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46923 (local address 127.0.98.62:38213)
0925 16:27:41.714439 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.714443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.714463 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:41.714703 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.714706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.714762 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.714858 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.715580 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.716131 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.717494 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.717740 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.717915 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.721703 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46923: BlockingRecv error: recv got EOF from 127.0.98.2:46923 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:41.730288 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.722662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:44203)
0925 16:27:41.722917 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.722932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.723027 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.723352 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.723356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.723549 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.723669 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.723674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.724856 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.724859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.725603 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.725618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.726393 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.726398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.726492 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.727218 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.727232 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.730123 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260925 16:27:41.730568 9423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:27:41.730731 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.722804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44203 (local address 127.0.98.62:38213)
0925 16:27:41.722977 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.722982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.723004 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:27:41.723132 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.723136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.723197 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.723305 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.723791 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.724689 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.725749 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.726198 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.726320 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.730334 (+ 4014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44203: BlockingRecv error: recv got EOF from 127.0.98.2:44203 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 16:27:41.854272 9426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:41.854713 9426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:41.854768 9426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:41.859622 9426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:41.859803 9426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:41.859894 9426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:41.867555 9426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37399
--tserver_master_addrs=127.0.98.62:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:41.869252 9426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:41.870711 9426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:41.884725 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.884725 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:41.884728 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:41.885380 9426 server_base.cc:1034] running on GCE node
I20260925 16:27:41.886001 9426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:41.886905 9426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:41.888258 9426 hybrid_clock.cc:648] HybridClock initialized: now 1790353661888230 us; error 63 us; skew 500 ppm
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353661, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:27:41.892405 9426 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 16:27:41.893939 9426 webserver.cc:460] Webserver started at http://127.0.98.3:37399/ using document root <none> and password file <none>
I20260925 16:27:41.894739 9426 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:41.894857 9426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:41.900718 9426 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 16:27:41.904474 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.906940 9426 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:27:41.907306 9426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "07d3abb5c65544fc89537717e31891d3"
format_stamp: "Formatted at 2026-09-25 16:27:40 on dist-test-slave-31kt"
server_key: "277cd0135e293f42b03103bb1ceb8501"
server_key_iv: "7273c1b96c44db3542286283c428801a"
server_key_version: "encryptionkey@0"
I20260925 16:27:41.908347 9426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:41.940104 9426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:41.944976 9426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:41.945230 9426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:41.946102 9426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:41.947296 9426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:41.947351 9426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.947407 9426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:41.947420 9426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:41.959326 9426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41629
I20260925 16:27:41.959349 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41629 every 8 connection(s)
I20260925 16:27:41.960809 9426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:41.963868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9426
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:41.978358 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.963414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51117 (local address 127.0.98.62:38213)
0925 16:27:41.963650 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.963656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.965214 (+ 1558us) server_negotiation.cc:407] Connection header received
0925 16:27:41.968267 (+ 3053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.968272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.968339 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.968440 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.970495 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.971349 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.973156 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.973497 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.973677 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.978187 (+ 4510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51117: BlockingRecv error: recv got EOF from 127.0.98.3:51117 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":111}
I20260925 16:27:41.979347 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.965338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.98.62:38213)
0925 16:27:41.966158 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.966162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.966184 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:27:41.966379 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.966383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.966451 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.966716 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.968267 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.968954 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.970495 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.970931 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.971078 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.973108 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:41.973178 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:41.973211 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:27:41.973275 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:41.975965 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:41.976754 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:41.976762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:41.976769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:41.977101 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:27:41.977467 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:27:41.977470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:27:41.977471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:27:41.978056 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:41.978273 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:41.978864 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 16:27:41.979200 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":667,"thread_start_us":155,"threads_started":1}
I20260925 16:27:41.979622 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.964048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:51117)
0925 16:27:41.964947 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.965039 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.967709 (+ 2670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.968812 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.968825 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.969391 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.970223 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.970249 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.971526 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.971529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.972999 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.973013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.973831 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.973837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.974100 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.974686 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.974710 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.978043 (+ 3333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":210,"threads_started":1}
W20260925 16:27:41.980545 9555 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38213 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38213: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:41.987933 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.981228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:44965)
0925 16:27:41.981381 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.981400 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.981516 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.981896 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.981900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.982129 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.982280 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.982287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.983240 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.983245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.983867 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.983879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.984422 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.984425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.984516 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.985118 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.985130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.987757 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 16:27:41.988266 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.981429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44965 (local address 127.0.98.62:38213)
0925 16:27:41.981596 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.981601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.981617 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:27:41.981697 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.981700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.981745 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.981862 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.982414 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.983114 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.984003 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.984310 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.984409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.988092 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44965: BlockingRecv error: recv got EOF from 127.0.98.3:44965 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:41 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:41.995390 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:41.988791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:58013)
0925 16:27:41.988993 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:41.989013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:41.989141 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:41.989586 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:41.989590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:41.989859 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:41.990006 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.990016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.991019 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.991023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.991659 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:41.991671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.992317 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.992320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:41.992424 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.993131 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:41.993144 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:41.995211 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 16:27:41.995735 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:41.988975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58013 (local address 127.0.98.62:38213)
0925 16:27:41.989236 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:41.989240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:41.989258 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:27:41.989309 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:41.989312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:41.989362 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:41.989444 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:41.990223 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.990852 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.991846 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:41.992136 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:41.992253 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:41.995397 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58013: BlockingRecv error: recv got EOF from 127.0.98.3:58013 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260925 16:27:41.995810 9555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 16:27:42 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:42 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:42.471115 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:42.464002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:53401)
0925 16:27:42.464312 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:42.464328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:42.464458 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:42.464876 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:42.464879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:42.465084 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:42.465250 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:42.465257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.466324 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.466329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:42.466960 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:42.466972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.467697 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.467702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:42.467844 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:42.468495 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:42.468509 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:42.470953 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
W20260925 16:27:42.471519 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:42.464161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53401 (local address 127.0.98.62:38213)
0925 16:27:42.464344 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:42.464348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:42.464415 (+ 67us) server_negotiation.cc:407] Connection header received
0925 16:27:42.464601 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:42.464604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:42.464667 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:42.464779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:42.465415 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.466083 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.467226 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.467471 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.467568 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:42.471293 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53401: BlockingRecv error: recv got EOF from 127.0.98.1:53401 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 16:27:42 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:42 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:42.738448 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:42.731560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:39311)
0925 16:27:42.731974 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:42.731997 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:42.732152 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:42.732576 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:42.732579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:42.732824 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:42.732981 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:42.732988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.733914 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.733917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:42.734534 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:42.734544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.735117 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.735121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:42.735236 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:42.735880 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:42.735895 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:42.738282 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
W20260925 16:27:42.738693 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:42.731695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39311 (local address 127.0.98.62:38213)
0925 16:27:42.732049 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:42.732056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:42.732091 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:27:42.732273 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:42.732276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:42.732337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:42.732437 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:42.733144 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.733735 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.734667 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.734909 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.735042 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:42.738517 (+ 3475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39311: BlockingRecv error: recv got EOF from 127.0.98.2:39311 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 16:27:42 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:42 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:43.004462 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:42.996794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:58233)
0925 16:27:42.997187 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:42.997205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:42.997342 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:42.997905 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:42.997908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:42.998193 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:42.998373 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:42.998381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.999380 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:42.999383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:43.000034 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:43.000045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.000788 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.000791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:43.000898 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:43.001563 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:43.001577 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:43.004229 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":176,"threads_started":1}
W20260925 16:27:43.004613 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:42.997060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58233 (local address 127.0.98.62:38213)
0925 16:27:42.997245 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:42.997249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:42.997273 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:27:42.997563 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:42.997568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:42.997663 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:42.997840 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:42.998578 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:42.999242 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.000246 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.000566 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.000756 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:43.004445 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58233: BlockingRecv error: recv got EOF from 127.0.98.3:58233 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 16:27:43 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:43 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:43.480185 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:43.472585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:44553)
0925 16:27:43.472919 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:43.472935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:43.473054 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:43.473605 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:43.473608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:43.473891 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:43.474018 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:43.474025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.475047 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.475052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:43.475720 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:43.475734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.476333 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.476336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:43.476451 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:43.477265 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:43.477283 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:43.479982 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
W20260925 16:27:43.480597 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:43.472784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44553 (local address 127.0.98.62:38213)
0925 16:27:43.473003 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:43.473007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:43.473027 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:43.473269 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:43.473273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:43.473345 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:43.473466 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:43.474206 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.474822 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.475891 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.476174 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.476277 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:43.480365 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44553: BlockingRecv error: recv got EOF from 127.0.98.1:44553 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 25 16:27:43 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:43 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:43.747681 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:43.739867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:43717)
0925 16:27:43.740406 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:43.740431 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:43.740648 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:43.740971 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:43.740974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:43.741260 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:43.741401 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:43.741408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.742371 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.742375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:43.743078 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:43.743091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.743957 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.743961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:43.744073 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:43.744871 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:43.744885 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:43.747489 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":112,"threads_started":1}
W20260925 16:27:43.747905 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:43.740238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43717 (local address 127.0.98.62:38213)
0925 16:27:43.740488 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:43.740493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:43.740550 (+ 57us) server_negotiation.cc:407] Connection header received
0925 16:27:43.740772 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:43.740776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:43.740836 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:43.740919 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:43.741570 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.742234 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.743388 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:43.743750 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:43.743909 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:43.747725 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43717: BlockingRecv error: recv got EOF from 127.0.98.2:43717 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 25 16:27:44 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:44 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:44.013229 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:44.005979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:49189)
0925 16:27:44.006332 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:44.006347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:44.006483 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:44.006913 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:44.006917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:44.007174 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:44.007316 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:44.007324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.008379 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.008382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:44.009018 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:44.009028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.009491 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.009494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:44.009579 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:44.010298 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:44.010311 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:44.012971 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
W20260925 16:27:44.013375 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:44.006215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49189 (local address 127.0.98.62:38213)
0925 16:27:44.006380 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:44.006384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:44.006411 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:27:44.006627 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:44.006630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:44.006698 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:44.006810 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:44.007492 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.008216 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.009138 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.009377 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.009474 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:44.013207 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49189: BlockingRecv error: recv got EOF from 127.0.98.3:49189 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 16:27:44 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:44 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:44.489477 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:44.481343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:53633)
0925 16:27:44.481685 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:44.481702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:44.481900 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:44.482329 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:44.482331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:44.482594 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:44.482840 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:44.482849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.483935 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.483938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:44.484644 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:44.484656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.485374 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.485381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:44.485594 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:44.486390 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:44.486404 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:44.489237 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
W20260925 16:27:44.489669 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:44.481651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53633 (local address 127.0.98.62:38213)
0925 16:27:44.481958 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:44.481964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:44.481985 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:27:44.482049 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:44.482051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:44.482108 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:44.482226 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:44.483049 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.483681 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.484802 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.485130 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.485269 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:44.489512 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53633: BlockingRecv error: recv got EOF from 127.0.98.1:53633 (error 108)
Metrics: {"server-negotiator.queue_time_us":194}
Sep 25 16:27:44 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:44 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:44.756551 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:44.749127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:36743)
0925 16:27:44.749622 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:44.749642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:44.749819 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:44.750181 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:44.750183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:44.750452 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:44.750591 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:44.750599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.751545 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.751548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:44.752216 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:44.752226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.752885 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.752888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:44.752969 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:44.753660 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:44.753673 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:44.756373 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":177,"threads_started":1}
W20260925 16:27:44.756920 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:44.749206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36743 (local address 127.0.98.62:38213)
0925 16:27:44.749350 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:44.749354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:44.749744 (+ 390us) server_negotiation.cc:407] Connection header received
0925 16:27:44.749936 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:44.749939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:44.750006 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:44.750125 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:44.750737 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.751357 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.752403 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:44.752668 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:44.752788 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:44.756724 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36743: BlockingRecv error: recv got EOF from 127.0.98.2:36743 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 16:27:45 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:45 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:45.026714 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:45.014709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:35197)
0925 16:27:45.015128 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:45.015147 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:45.015327 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:45.015750 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:45.015753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:45.016025 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:45.016176 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:45.016183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.017112 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.017115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:45.017833 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:45.017847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.018452 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.018455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:45.018557 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:45.023478 (+ 4921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:45.023501 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:45.026475 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":103,"threads_started":1}
W20260925 16:27:45.027022 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:45.014792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35197 (local address 127.0.98.62:38213)
0925 16:27:45.015075 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:45.015081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:45.015290 (+ 209us) server_negotiation.cc:407] Connection header received
0925 16:27:45.015502 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:45.015505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:45.015592 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:45.015692 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:45.016346 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.016938 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.018012 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.018268 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.018388 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:45.026776 (+ 8388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35197: BlockingRecv error: recv got EOF from 127.0.98.3:35197 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 25 16:27:45 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:45 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:45.498576 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:45.491056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:34545)
0925 16:27:45.491480 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:45.491499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:45.491661 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:45.492044 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:45.492048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:45.492309 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:45.492452 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:45.492460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.493433 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.493438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:45.494110 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:45.494123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.494809 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.494814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:45.494933 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:45.495668 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:45.495682 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:45.498415 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
W20260925 16:27:45.498783 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:45.491195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34545 (local address 127.0.98.62:38213)
0925 16:27:45.491434 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:45.491439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:45.491614 (+ 175us) server_negotiation.cc:407] Connection header received
0925 16:27:45.491767 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:45.491770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:45.491830 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:45.491931 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:45.492646 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.493249 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.494306 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.494558 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.494668 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:45.498608 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34545: BlockingRecv error: recv got EOF from 127.0.98.1:34545 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 16:27:45 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:45 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:45.765120 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:45.758044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:39929)
0925 16:27:45.758400 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:45.758415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:45.758563 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:45.759068 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:45.759073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:45.759284 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:45.759419 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:45.759427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.760419 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.760422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:45.761091 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:45.761105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.761650 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.761653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:45.761736 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:45.762487 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:45.762501 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:45.764949 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":130,"threads_started":1}
W20260925 16:27:45.765504 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:45.758178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39929 (local address 127.0.98.62:38213)
0925 16:27:45.758366 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:45.758372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:45.758527 (+ 155us) server_negotiation.cc:407] Connection header received
0925 16:27:45.758690 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:45.758694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:45.758798 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:45.758886 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:45.759606 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.760237 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.761256 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:45.761502 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:45.761620 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:45.765306 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39929: BlockingRecv error: recv got EOF from 127.0.98.2:39929 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 16:27:46 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:46 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:46.035043 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:46.028057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:52303)
0925 16:27:46.028488 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:46.028506 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:46.028683 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:46.028969 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:46.028972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:46.029214 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:46.029324 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:46.029332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.030263 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.030269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:46.030982 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:46.030995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.031485 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.031489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:46.031568 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:46.032270 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:46.032288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:46.034801 (+ 2513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":155,"threads_started":1}
W20260925 16:27:46.035286 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:46.028158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52303 (local address 127.0.98.62:38213)
0925 16:27:46.028419 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:46.028425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:46.028614 (+ 189us) server_negotiation.cc:407] Connection header received
0925 16:27:46.028789 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:46.028792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:46.028846 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:46.028919 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:46.029451 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.030079 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.031140 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.031360 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.031454 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:46.035120 (+ 3666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52303: BlockingRecv error: recv got EOF from 127.0.98.3:52303 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 25 16:27:46 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:46 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:46.508016 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:46.499810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:43993)
0925 16:27:46.500206 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:46.500222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:46.500367 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:46.500865 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:46.500868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:46.501109 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:46.501311 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:46.501319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.502347 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.502393 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:46.503129 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:46.503144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.503858 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.503863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:46.504024 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:46.504734 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:46.504748 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:46.507777 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
W20260925 16:27:46.508041 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:46.499923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43993 (local address 127.0.98.62:38213)
0925 16:27:46.500121 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:46.500126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:46.500304 (+ 178us) server_negotiation.cc:407] Connection header received
0925 16:27:46.500577 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:46.500581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:46.500656 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:46.500796 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:46.501478 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.502156 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.503333 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.503602 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.503750 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:46.507908 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43993: BlockingRecv error: recv got EOF from 127.0.98.1:43993 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 16:27:46 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:46 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:46.773808 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:46.766346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:40783)
0925 16:27:46.766756 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:46.766777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:46.766939 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:46.767292 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:46.767296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:46.767534 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:46.767710 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:46.767717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.768549 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.768553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:46.769209 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:46.769220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.769711 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.769714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:46.769837 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:46.770742 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:46.770757 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:46.773599 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
W20260925 16:27:46.774104 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:46.766606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40783 (local address 127.0.98.62:38213)
0925 16:27:46.766790 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:46.766795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:46.766874 (+ 79us) server_negotiation.cc:407] Connection header received
0925 16:27:46.767060 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:46.767062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:46.767161 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:46.767246 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:46.767842 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.768433 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.769348 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:46.769586 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:46.769687 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:46.773952 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40783: BlockingRecv error: recv got EOF from 127.0.98.2:40783 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 25 16:27:47 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:47 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:47.043186 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:47.036132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:36647)
0925 16:27:47.036471 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:47.036486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:47.036653 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:47.036990 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:47.036992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:47.037216 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:47.037335 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:47.037341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.038327 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.038329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:47.038994 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:47.039003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.039590 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.039593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:47.039716 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:47.040334 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:47.040347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:47.042957 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
W20260925 16:27:47.043289 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:47.036230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36647 (local address 127.0.98.62:38213)
0925 16:27:47.036454 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:47.036458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:47.036616 (+ 158us) server_negotiation.cc:407] Connection header received
0925 16:27:47.036774 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:47.036777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:47.036833 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:47.036916 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:47.037517 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.038159 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.039139 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.039422 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.039542 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:47.043145 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36647: BlockingRecv error: recv got EOF from 127.0.98.3:36647 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 25 16:27:47 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:47 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:47.516665 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:47.509272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:56491)
0925 16:27:47.509670 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:47.509693 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:47.509918 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:47.510344 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:47.510347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:47.510593 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:47.510728 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:47.510735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.511692 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.511695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:47.512311 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:47.512320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.512969 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.512973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:47.513064 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:47.513885 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:47.513903 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:47.516515 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
W20260925 16:27:47.516884 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:47.509467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56491 (local address 127.0.98.62:38213)
0925 16:27:47.509718 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:47.509725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:47.509830 (+ 105us) server_negotiation.cc:407] Connection header received
0925 16:27:47.510089 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:47.510094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:47.510174 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:47.510262 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:47.510916 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.511514 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.512503 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.512779 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.512916 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:47.516720 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56491: BlockingRecv error: recv got EOF from 127.0.98.1:56491 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 25 16:27:47 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:47 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:47.781849 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:47.774774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:50559)
0925 16:27:47.775052 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:47.775066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:47.775173 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:47.775734 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:47.775736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:47.775939 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:47.776068 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:47.776074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.777235 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.777238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:47.777935 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:47.777947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.778551 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.778554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:47.778645 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:47.779330 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:47.779349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:47.781670 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
W20260925 16:27:47.782148 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:47.774973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50559 (local address 127.0.98.62:38213)
0925 16:27:47.775135 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:47.775139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:47.775157 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:27:47.775347 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:47.775350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:47.775432 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:47.775555 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:47.776272 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.776995 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.778124 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:47.778378 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:47.778503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:47.781973 (+ 3470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50559: BlockingRecv error: recv got EOF from 127.0.98.2:50559 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 16:27:48 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:48 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:48.051658 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:48.044447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:34889)
0925 16:27:48.044720 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:48.044736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:48.044845 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:48.045269 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:48.045273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:48.045473 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:48.045599 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:48.045607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.046586 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.046588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:48.047165 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:48.047172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.047904 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.047910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:48.047997 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:48.048812 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:48.048828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:48.051438 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
W20260925 16:27:48.051900 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:48.044669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34889 (local address 127.0.98.62:38213)
0925 16:27:48.044828 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:48.044832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:48.044853 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:27:48.044960 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:48.044963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:48.045052 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:48.045164 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:48.045852 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.046428 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.047341 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.047657 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.047830 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:48.051742 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34889: BlockingRecv error: recv got EOF from 127.0.98.3:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 16:27:48 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:48 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:48.526794 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:48.517898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:37841)
0925 16:27:48.518561 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:48.518585 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:48.518862 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:48.519457 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:48.519467 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:48.519842 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:48.520000 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:48.520010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.520995 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.520999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:48.521652 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:48.521663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.522360 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.522365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:48.522498 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:48.523228 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:48.523277 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:48.526591 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":211,"threads_started":1}
W20260925 16:27:48.531159 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:48.518341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37841 (local address 127.0.98.62:38213)
0925 16:27:48.518583 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:48.518791 (+ 208us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:48.518820 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:27:48.519096 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:48.519100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:48.519223 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:48.519367 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:48.520169 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.520834 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.521822 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.522102 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.522203 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:48.530940 (+ 8737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37841: BlockingRecv error: recv got EOF from 127.0.98.1:37841 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 25 16:27:48 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:48 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:48.790958 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:48.783263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43941 (local address 127.0.98.62:38213)
0925 16:27:48.783524 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:48.783530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:48.783723 (+ 193us) server_negotiation.cc:407] Connection header received
0925 16:27:48.783914 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:48.783918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:48.783989 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:48.784067 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:48.784581 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.785205 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.786121 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.786356 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.786452 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:48.790791 (+ 4339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43941: BlockingRecv error: recv got EOF from 127.0.98.2:43941 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 16:27:48.791345 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:48.783054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:43941)
0925 16:27:48.783606 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:48.783624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:48.783786 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:48.784115 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:48.784118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:48.784334 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:48.784448 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:48.784454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.785327 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.785331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:48.785993 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:48.786005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:48.786477 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:48.786481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:48.786557 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:48.787267 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:48.787313 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:48.791183 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":127,"threads_started":1}
Sep 25 16:27:49 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:49 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:49.063902 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:49.052892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:38807)
0925 16:27:49.053402 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:49.053417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:49.053555 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:49.053847 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:49.053850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:49.054085 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:49.054188 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:49.054195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.054974 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.054978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:49.055564 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:49.055573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.056059 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.056063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:49.056148 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:49.060513 (+ 4365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:49.060532 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:49.063738 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":132,"threads_started":1}
W20260925 16:27:49.064392 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:49.053014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38807 (local address 127.0.98.62:38213)
0925 16:27:49.053310 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:49.053315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:49.053506 (+ 191us) server_negotiation.cc:407] Connection header received
0925 16:27:49.053653 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:49.053656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:49.053703 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:49.053818 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:49.054326 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.054860 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.055684 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.055935 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.056026 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:49.064196 (+ 8170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38807: BlockingRecv error: recv got EOF from 127.0.98.3:38807 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
Sep 25 16:27:49 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:49 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:49.537837 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:49.530062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:46735)
0925 16:27:49.530475 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:49.530494 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:49.530610 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:49.531098 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:49.531102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:49.531360 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:49.531518 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:49.531526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.532677 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.532681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:49.533365 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:49.533379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.533913 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.533917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:49.534007 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:49.534752 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:49.534766 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:49.537512 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":91,"threads_started":1}
W20260925 16:27:49.537921 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:49.530475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46735 (local address 127.0.98.62:38213)
0925 16:27:49.530688 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:49.530693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:49.530713 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:49.530781 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:49.530784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:49.530843 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:49.530956 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:49.531812 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.532522 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.533499 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.533743 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.533879 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:49.537789 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46735: BlockingRecv error: recv got EOF from 127.0.98.1:46735 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 25 16:27:49 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:49 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:49.801129 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:49.792797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:42253)
0925 16:27:49.793202 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:49.793224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:49.793389 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:49.793822 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:49.793826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:49.794023 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:49.794149 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:49.794154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.795321 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.795325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:49.795971 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:49.795983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.796873 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.796876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:49.797003 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:49.797996 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:49.798018 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:49.800918 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":104,"threads_started":1}
W20260925 16:27:49.801285 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:49.793038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42253 (local address 127.0.98.62:38213)
0925 16:27:49.793237 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:49.793242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:49.793335 (+ 93us) server_negotiation.cc:407] Connection header received
0925 16:27:49.793499 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:49.793502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:49.793563 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:49.793657 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:49.794352 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.795133 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.796196 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:49.796673 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:49.796813 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:49.801146 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42253: BlockingRecv error: recv got EOF from 127.0.98.2:42253 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 16:27:50 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:50 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:50.078061 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:50.065097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:51191)
0925 16:27:50.065519 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:50.065538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:50.065724 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:50.066105 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:50.066108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:50.066401 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:50.066537 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:50.066545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.067648 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.067652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:50.068316 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:50.068330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.069025 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.069028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:50.069113 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:50.075465 (+ 6352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:50.075486 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:50.077912 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":89,"threads_started":1}
W20260925 16:27:50.078459 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:50.065413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51191 (local address 127.0.98.62:38213)
0925 16:27:50.065570 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:50.065574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:50.065726 (+ 152us) server_negotiation.cc:407] Connection header received
0925 16:27:50.065861 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:50.065865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:50.065931 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:50.066026 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:50.066752 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.067373 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.068581 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.068863 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.068976 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:50.078232 (+ 9256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51191: BlockingRecv error: recv got EOF from 127.0.98.3:51191 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 16:27:50 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:50 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:50.546891 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:50.539230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:33043)
0925 16:27:50.539727 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:50.539747 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:50.539970 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:50.540299 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:50.540302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:50.540593 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:50.540735 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:50.540744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.541645 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.541650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:50.542358 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:50.542373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.542935 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.542938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:50.543028 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:50.543766 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:50.543780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:50.546699 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":196,"threads_started":1}
W20260925 16:27:50.547147 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:50.539459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33043 (local address 127.0.98.62:38213)
0925 16:27:50.539684 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:50.539691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:50.539858 (+ 167us) server_negotiation.cc:407] Connection header received
0925 16:27:50.540104 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:50.540108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:50.540169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:50.540252 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:50.540897 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.541505 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.542514 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.542789 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.542895 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:50.546937 (+ 4042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33043: BlockingRecv error: recv got EOF from 127.0.98.1:33043 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 16:27:50 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:50 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:50.809904 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:50.802332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:60643)
0925 16:27:50.802639 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:50.802654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:50.802752 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:50.803093 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:50.803116 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:50.803309 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:50.803432 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:50.803438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.804403 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.804406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:50.805165 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:50.805178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.805896 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.805899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:50.805992 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:50.806731 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:50.806744 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:50.809691 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
W20260925 16:27:50.810295 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:50.802483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60643 (local address 127.0.98.62:38213)
0925 16:27:50.802676 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:50.802680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:50.802700 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:27:50.802858 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:50.802861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:50.802923 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:50.803030 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:50.803545 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.804240 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.805379 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:50.805658 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:50.805801 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:50.810110 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60643: BlockingRecv error: recv got EOF from 127.0.98.2:60643 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 16:27:51 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:51 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:51.086553 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:51.079249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:52529)
0925 16:27:51.079602 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:51.079618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:51.079752 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:51.080195 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:51.080199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:51.080458 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:51.080656 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:51.080664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.081641 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.081644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:51.082275 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:51.082286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.082740 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.082743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:51.082819 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:51.083529 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:51.083547 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:51.086382 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
W20260925 16:27:51.086786 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:51.079418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52529 (local address 127.0.98.62:38213)
0925 16:27:51.079596 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:51.079601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:51.079703 (+ 102us) server_negotiation.cc:407] Connection header received
0925 16:27:51.079857 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:51.079862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:51.079951 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:51.080051 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:51.080830 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.081517 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.082400 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.082622 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.082717 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:51.086643 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52529: BlockingRecv error: recv got EOF from 127.0.98.3:52529 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 16:27:51 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:51 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:51.555943 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:51.548154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:36309)
0925 16:27:51.548712 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:51.548733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:51.548900 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:51.549212 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:51.549214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:51.549429 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:51.549532 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:51.549538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.550581 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.550584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:51.551332 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:51.551346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.551858 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.551862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:51.551947 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:51.552837 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:51.552858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:51.555769 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":295,"threads_started":1}
W20260925 16:27:51.556257 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:51.548396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36309 (local address 127.0.98.62:38213)
0925 16:27:51.548712 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:51.548717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:51.548835 (+ 118us) server_negotiation.cc:407] Connection header received
0925 16:27:51.549016 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:51.549019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:51.549071 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:51.549146 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:51.549674 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.550250 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.551475 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.551711 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.551809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:51.556069 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36309: BlockingRecv error: recv got EOF from 127.0.98.1:36309 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Sep 25 16:27:51 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:51 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:51.819008 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:51.811303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:54595)
0925 16:27:51.811807 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:51.811832 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:51.812025 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:51.812356 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:51.812359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:51.812614 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:51.812740 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:51.812747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.813688 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.813692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:51.814416 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:51.814429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.814966 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.814969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:51.815052 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:51.815903 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:51.815963 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:51.818813 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":195,"threads_started":1}
W20260925 16:27:51.819253 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:51.811621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54595 (local address 127.0.98.62:38213)
0925 16:27:51.811844 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:51.811850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:51.811952 (+ 102us) server_negotiation.cc:407] Connection header received
0925 16:27:51.812134 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:51.812138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:51.812191 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:51.812267 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:51.812872 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.813561 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.814587 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:51.814824 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:51.814929 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:51.819066 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54595: BlockingRecv error: recv got EOF from 127.0.98.2:54595 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 16:27:52 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:52 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:52.095880 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:52.087814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:52891)
0925 16:27:52.088309 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:52.088325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:52.088483 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:52.088930 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:52.088933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:52.089177 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:52.089324 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:52.089331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.090523 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.090528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:52.091356 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:52.091373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.091986 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.091989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:52.092067 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:52.092755 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:52.092770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:52.095664 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":131,"threads_started":1}
W20260925 16:27:52.096063 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:52.088108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52891 (local address 127.0.98.62:38213)
0925 16:27:52.088328 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:52.088333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:52.088472 (+ 139us) server_negotiation.cc:407] Connection header received
0925 16:27:52.088598 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:52.088600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:52.088659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:52.088736 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:52.089529 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.090282 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.091537 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.091843 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.091955 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:52.095911 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52891: BlockingRecv error: recv got EOF from 127.0.98.3:52891 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 25 16:27:52 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:52 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:52.564093 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:52.557284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:42861)
0925 16:27:52.557636 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:52.557653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:52.557839 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:52.558213 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:52.558216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:52.558515 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:52.558658 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:52.558666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.559603 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.559608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:52.560318 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:52.560329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.560787 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.560790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:52.560874 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:52.561554 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:52.561568 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:52.563932 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
W20260925 16:27:52.564373 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:52.557480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42861 (local address 127.0.98.62:38213)
0925 16:27:52.557646 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:52.557650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:52.557787 (+ 137us) server_negotiation.cc:407] Connection header received
0925 16:27:52.557956 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:52.557960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:52.558033 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:52.558163 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:52.558848 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.559491 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.560439 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.560676 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.560772 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:52.564182 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42861: BlockingRecv error: recv got EOF from 127.0.98.1:42861 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 16:27:52 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:52 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:52.827922 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:52.820246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:60279)
0925 16:27:52.820608 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:52.820627 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:52.820780 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:52.821213 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:52.821217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:52.821446 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:52.821588 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:52.821595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.822690 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.822693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:52.823527 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:52.823549 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.824204 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.824208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:52.824288 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:52.825078 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:52.825096 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:52.827585 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":128,"threads_started":1}
W20260925 16:27:52.827996 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:52.820353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60279 (local address 127.0.98.62:38213)
0925 16:27:52.820542 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:52.820548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:52.820781 (+ 233us) server_negotiation.cc:407] Connection header received
0925 16:27:52.820876 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:52.820879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:52.820940 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:52.821029 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:52.821844 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.822519 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.823766 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:52.824038 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:52.824157 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:52.827857 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60279: BlockingRecv error: recv got EOF from 127.0.98.2:60279 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 16:27:53 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:53 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:53.104013 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:53.097096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:40763)
0925 16:27:53.097478 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:53.097497 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:53.097656 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:53.098003 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:53.098006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:53.098256 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:53.098378 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:53.098385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.099210 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.099214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:53.099826 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:53.099836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.100306 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.100310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:53.100394 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:53.101117 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:53.101129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:53.103763 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
W20260925 16:27:53.104135 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:53.097279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40763 (local address 127.0.98.62:38213)
0925 16:27:53.097479 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:53.097485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:53.097590 (+ 105us) server_negotiation.cc:407] Connection header received
0925 16:27:53.097824 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:53.097828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:53.097882 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:53.097962 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:53.098508 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.099095 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.099958 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.100183 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.100284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:53.104004 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40763: BlockingRecv error: recv got EOF from 127.0.98.3:40763 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 16:27:53 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:53 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:53.573132 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:53.565197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:43051)
0925 16:27:53.565695 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:53.565715 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:53.565954 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:53.566356 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:53.566359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:53.566685 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:53.566826 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:53.566834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.567892 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.567895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:53.568562 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:53.568570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.569169 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.569172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:53.569249 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:53.570250 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:53.570264 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:53.572943 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":95,"threads_started":1}
W20260925 16:27:53.573227 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:53.565480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43051 (local address 127.0.98.62:38213)
0925 16:27:53.565662 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:53.565667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:53.565964 (+ 297us) server_negotiation.cc:407] Connection header received
0925 16:27:53.566068 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:53.566071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:53.566149 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:53.566317 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:53.567029 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.567753 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.568745 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.569035 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.569137 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:53.573114 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43051: BlockingRecv error: recv got EOF from 127.0.98.1:43051 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 16:27:53 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:53 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:53.837185 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:53.829154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:38463)
0925 16:27:53.829571 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:53.829592 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:53.829825 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:53.830192 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:53.830196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:53.830457 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:53.830623 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:53.830632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.831813 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.831818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:53.832714 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:53.832733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.833439 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.833443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:53.833717 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:53.834578 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:53.834595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:53.836991 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":120,"threads_started":1}
W20260925 16:27:53.837423 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:53.829335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38463 (local address 127.0.98.62:38213)
0925 16:27:53.829571 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:53.829577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:53.829712 (+ 135us) server_negotiation.cc:407] Connection header received
0925 16:27:53.829948 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:53.829952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:53.830018 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:53.830140 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:53.830896 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.831593 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.832899 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:53.833244 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:53.833413 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:53.837257 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38463: BlockingRecv error: recv got EOF from 127.0.98.2:38463 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 16:27:54 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:54 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:54.112881 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:54.105548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:47309)
0925 16:27:54.105935 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:54.105953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:54.106125 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:54.106443 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:54.106446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:54.106731 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:54.106857 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:54.106864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.107812 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.107816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:54.108545 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:54.108558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.109064 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.109068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:54.109157 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:54.109858 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:54.109873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:54.112734 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":115,"threads_started":1}
W20260925 16:27:54.113291 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:54.105710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47309 (local address 127.0.98.62:38213)
0925 16:27:54.105960 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:54.105965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:54.106050 (+ 85us) server_negotiation.cc:407] Connection header received
0925 16:27:54.106254 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:54.106257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:54.106323 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:54.106399 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:54.106982 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.107635 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.108705 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.108948 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.109044 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:54.113066 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47309: BlockingRecv error: recv got EOF from 127.0.98.3:47309 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 16:27:54 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:54 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:54.583069 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:54.574679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:41141)
0925 16:27:54.575061 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:54.575078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:54.575239 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:54.575659 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:54.575661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:54.575901 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:54.576041 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:54.576047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.577224 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.577229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:54.578188 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:54.578205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.578947 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.578952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:54.579125 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:54.579950 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:54.579969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:54.582878 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
W20260925 16:27:54.583310 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:54.574746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41141 (local address 127.0.98.62:38213)
0925 16:27:54.575000 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:54.575006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:54.575218 (+ 212us) server_negotiation.cc:407] Connection header received
0925 16:27:54.575435 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:54.575438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:54.575497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:54.575588 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:54.576323 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.576939 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.578430 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.578746 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.578871 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:54.583156 (+ 4285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41141: BlockingRecv error: recv got EOF from 127.0.98.1:41141 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 25 16:27:54 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:54 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:54.845136 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:54.838268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:39721)
0925 16:27:54.838658 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:54.838672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:54.838794 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:54.839108 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:54.839111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:54.839324 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:54.839500 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:54.839506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.840397 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.840400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:54.841038 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:54.841048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.841882 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.841885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:54.841969 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:54.842531 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:54.842543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:54.844920 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":85,"threads_started":1}
W20260925 16:27:54.845269 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:54.838483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39721 (local address 127.0.98.62:38213)
0925 16:27:54.838663 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:54.838668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:54.838760 (+ 92us) server_negotiation.cc:407] Connection header received
0925 16:27:54.838898 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:54.838902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:54.838964 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:54.839042 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:54.839612 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.840262 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.841394 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:54.841662 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:54.841794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:54.845134 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39721: BlockingRecv error: recv got EOF from 127.0.98.2:39721 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 16:27:55 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:55 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:55.130357 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:55.113999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:36621)
0925 16:27:55.114449 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:55.114466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:55.114637 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:55.114985 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:55.114988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:55.115200 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:55.115315 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:55.115322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.116132 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.116136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:55.116716 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:55.116725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.117181 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.117185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:55.117264 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:55.127595 (+ 10331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:55.127612 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:55.130178 (+ 2566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":130,"threads_started":1}
W20260925 16:27:55.130498 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:55.114151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36621 (local address 127.0.98.62:38213)
0925 16:27:55.114390 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:55.114396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:55.114574 (+ 178us) server_negotiation.cc:407] Connection header received
0925 16:27:55.114772 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:55.114776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:55.114855 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:55.114932 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:55.115438 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.116015 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.116839 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.117059 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.117152 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:55.130374 (+ 13222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36621: BlockingRecv error: recv got EOF from 127.0.98.3:36621 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 16:27:55 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:55 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:55.591214 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:55.584302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:40339)
0925 16:27:55.584711 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:55.584729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:55.584878 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:55.585207 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:55.585210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:55.585432 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:55.585565 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:55.585572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.586624 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.586629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:55.587329 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:55.587342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.587909 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.587913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:55.588014 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:55.588755 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:55.588770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:55.591021 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
W20260925 16:27:55.591492 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:55.584474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40339 (local address 127.0.98.62:38213)
0925 16:27:55.584822 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:55.584828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:55.584850 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:27:55.584979 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:55.584981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:55.585049 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:55.585132 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:55.585727 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.586498 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.587449 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.587733 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.587853 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:55.591324 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40339: BlockingRecv error: recv got EOF from 127.0.98.1:40339 (error 108)
Metrics: {"server-negotiator.queue_time_us":222}
Sep 25 16:27:55 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:55 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 16:27:55.853902 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:55.846672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55757 (local address 127.0.98.62:38213)
0925 16:27:55.846868 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:55.846875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:55.847035 (+ 160us) server_negotiation.cc:407] Connection header received
0925 16:27:55.847233 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:55.847237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:55.847307 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:55.847398 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:55.848028 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.848641 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.849576 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.849897 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.850019 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:55.853745 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55757: BlockingRecv error: recv got EOF from 127.0.98.2:55757 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 16:27:55.854061 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:55.846472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:55757)
0925 16:27:55.846860 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:55.846884 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:55.847119 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:55.847441 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:55.847445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:55.847729 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:55.847867 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:55.847875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.848784 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.848788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:55.849421 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:55.849434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:55.850042 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:55.850047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:55.850139 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:55.850928 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:55.850942 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:55.853569 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
Sep 25 16:27:56 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:56 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:56.139169 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:56.131503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.3:59735)
0925 16:27:56.131967 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:56.131988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:56.132185 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:56.132469 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:56.132472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:56.132760 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:56.132885 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:56.132892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.133818 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.133822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:56.134624 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:56.134639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.135174 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.135178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:56.135274 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:56.135954 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:56.136008 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:56.138944 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":129,"threads_started":1}
W20260925 16:27:56.139452 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:56.131765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59735 (local address 127.0.98.62:38213)
0925 16:27:56.131967 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:56.131973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:56.132096 (+ 123us) server_negotiation.cc:407] Connection header received
0925 16:27:56.132298 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:56.132300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:56.132352 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:56.132428 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:56.133026 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.133608 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.134789 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.135038 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.135146 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:56.139260 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59735: BlockingRecv error: recv got EOF from 127.0.98.3:59735 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 16:27:56 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:56 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:56.599659 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:56.592772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.1:51913)
0925 16:27:56.593196 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:56.593220 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:56.593557 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:56.593926 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:56.593929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:56.594156 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:56.594264 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:56.594272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.595052 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.595056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:56.595635 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:56.595643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.596154 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.596158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:56.596243 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:56.596890 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:56.596903 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:56.599413 (+ 2510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":120,"threads_started":1}
W20260925 16:27:56.599821 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:56.592938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51913 (local address 127.0.98.62:38213)
0925 16:27:56.593220 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:56.593226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:56.593506 (+ 280us) server_negotiation.cc:407] Connection header received
0925 16:27:56.593698 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:56.593702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:56.593790 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:56.593872 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:56.594393 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.594932 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.595748 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.595997 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.596102 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:56.599634 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51913: BlockingRecv error: recv got EOF from 127.0.98.1:51913 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 16:27:56 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 16:27:56 dist-test-slave-31kt krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 16:27:56.862879 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:56.855410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38213 (local address 127.0.98.2:59399)
0925 16:27:56.855795 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:56.855814 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:56.855995 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:56.856268 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:56.856271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:56.856528 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:56.856650 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:56.856656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.857482 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.857485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:56.858364 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:56.858381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.858926 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.858930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:56.859009 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:56.859839 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:27:56.859856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:56.862717 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38213: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
W20260925 16:27:56.863235 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:56.855579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59399 (local address 127.0.98.62:38213)
0925 16:27:56.855797 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:56.855802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:56.855901 (+ 99us) server_negotiation.cc:407] Connection header received
0925 16:27:56.856104 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:56.856108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:56.856160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:56.856232 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:56.856781 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.857363 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.858529 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:56.858793 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:56.858911 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:56.863056 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59399: BlockingRecv error: recv got EOF from 127.0.98.2:59399 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 16:27:56.964331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9142
I20260925 16:27:56.971722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9294
I20260925 16:27:56.978511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9426
I20260925 16:27:56.985405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9072
2026-09-25T16:27:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18530 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T16:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:57Z Disabled control of system clock
I20260925 16:27:57.022255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39767
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 16:27:57.174017 9607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:57.174472 9607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:57.174517 9607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:57.179548 9607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:57.179693 9607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:57.179709 9607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:57.179728 9607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:57.179744 9607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:57.186501 9607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:57.188172 9607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:57.190207 9607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:57.201829 9612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:57.201829 9613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:57.202008 9615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:57.202194 9607 server_base.cc:1034] running on GCE node
I20260925 16:27:57.203229 9607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:57.205004 9607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:57.208408 9607 hybrid_clock.cc:648] HybridClock initialized: now 1790353677208359 us; error 50 us; skew 500 ppm
I20260925 16:27:57.211599 9607 webserver.cc:460] Webserver started at http://127.0.98.62:38303/ using document root <none> and password file <none>
I20260925 16:27:57.212464 9607 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:57.212564 9607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:57.212839 9607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:57.215878 9607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "e87db16426dd4ff4b91a74f99d5de83f"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "b516345e8a3867c07fb841c86646bdaf"
server_key_iv: "9e20e18b230aab3f2f65c88500f2f7ab"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.216737 9607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "e87db16426dd4ff4b91a74f99d5de83f"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "b516345e8a3867c07fb841c86646bdaf"
server_key_iv: "9e20e18b230aab3f2f65c88500f2f7ab"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.222116 9607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 16:27:57.226049 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.228015 9607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:57.228241 9607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "e87db16426dd4ff4b91a74f99d5de83f"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "b516345e8a3867c07fb841c86646bdaf"
server_key_iv: "9e20e18b230aab3f2f65c88500f2f7ab"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.228422 9607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:57.255421 9607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:57.256512 9607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:57.256762 9607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:57.266552 9607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39767
I20260925 16:27:57.266590 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39767 every 8 connection(s)
I20260925 16:27:57.267935 9607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:57.272327 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:57.272701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9607
I20260925 16:27:57.272823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:57.273166 392 external_mini_cluster.cc:1428] Setting key 9f3c1e74a0124dea55926be24c6c9785
I20260925 16:27:57.279553 9674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f: Bootstrap starting.
I20260925 16:27:57.283561 9674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:57.284765 9674 log.cc:824] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:57.286470 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:57.275138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.0.98.62:39767)
0925 16:27:57.276089 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:57.276110 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:57.276164 (+ 54us) server_negotiation.cc:407] Connection header received
0925 16:27:57.277056 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:57.277082 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:57.277647 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:57.278123 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:57.278622 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.279765 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.281289 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.282142 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.282318 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:57.283507 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:57.283514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:57.283538 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:57.284196 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:57.284301 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:57.284332 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:57.284436 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:57.284722 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 16:27:57.285140 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":192,"threads_started":1}
I20260925 16:27:57.288414 9674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f: No bootstrap required, opened a new log
I20260925 16:27:57.292897 9674 raft_consensus.cc:348] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87db16426dd4ff4b91a74f99d5de83f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39767 } }
I20260925 16:27:57.293540 9674 raft_consensus.cc:374] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:57.293607 9674 raft_consensus.cc:729] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87db16426dd4ff4b91a74f99d5de83f, State: Initialized, Role: FOLLOWER
I20260925 16:27:57.294342 9674 consensus_queue.cc:260] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87db16426dd4ff4b91a74f99d5de83f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39767 } }
I20260925 16:27:57.294530 9674 raft_consensus.cc:388] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:57.294589 9674 raft_consensus.cc:482] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:57.294687 9674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:57.296068 9674 raft_consensus.cc:504] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87db16426dd4ff4b91a74f99d5de83f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39767 } }
I20260925 16:27:57.296553 9674 leader_election.cc:302] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e87db16426dd4ff4b91a74f99d5de83f; no voters:
I20260925 16:27:57.296939 9674 leader_election.cc:288] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:57.297216 9679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:57.297597 9679 raft_consensus.cc:686] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [term 1 LEADER]: Becoming Leader. State: Replica: e87db16426dd4ff4b91a74f99d5de83f, State: Running, Role: LEADER
I20260925 16:27:57.298103 9679 consensus_queue.cc:237] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87db16426dd4ff4b91a74f99d5de83f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39767 } }
I20260925 16:27:57.298604 9674 sys_catalog.cc:564] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:57.300347 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e87db16426dd4ff4b91a74f99d5de83f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87db16426dd4ff4b91a74f99d5de83f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39767 } } }
I20260925 16:27:57.300576 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:57.300994 9681 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e87db16426dd4ff4b91a74f99d5de83f. Latest consensus state: current_term: 1 leader_uuid: "e87db16426dd4ff4b91a74f99d5de83f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87db16426dd4ff4b91a74f99d5de83f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39767 } } }
I20260925 16:27:57.301124 9681 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:57.301188 9690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:57.305253 9690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:57.314219 9690 catalog_manager.cc:1347] Generated new cluster ID: 2335eb4535c74c93b87f3979253c7d00
I20260925 16:27:57.314365 9690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:57.327679 9690 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:57.328828 9690 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:57.339490 9690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e87db16426dd4ff4b91a74f99d5de83f: Generated new TSK 0
I20260925 16:27:57.340452 9690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:57.347597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39767
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 16:27:57.507041 9698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:57.507473 9698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:57.507521 9698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:57.512353 9698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:57.512462 9698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:57.512530 9698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:57.518280 9698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:57.519894 9698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:57.521788 9698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:57.535104 9706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:57.535090 9703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:57.535090 9704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:57.535476 9698 server_base.cc:1034] running on GCE node
I20260925 16:27:57.536211 9698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:57.537163 9698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:57.540530 9698 hybrid_clock.cc:648] HybridClock initialized: now 1790353677540495 us; error 83 us; skew 500 ppm
I20260925 16:27:57.543722 9698 webserver.cc:460] Webserver started at http://127.0.98.1:45255/ using document root <none> and password file <none>
I20260925 16:27:57.544595 9698 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:57.544697 9698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:57.544975 9698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:57.547996 9698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "0fd1f69a229d4b58851eabd27afa70dc"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "a6a0eaf76e8df0ab68eb55116e2debed"
server_key_iv: "8aa25d28f7ac97fe30a97b65ee07fa37"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.548730 9698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0fd1f69a229d4b58851eabd27afa70dc"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "a6a0eaf76e8df0ab68eb55116e2debed"
server_key_iv: "8aa25d28f7ac97fe30a97b65ee07fa37"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.554206 9698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:27:57.558522 9712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.560575 9698 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:57.560837 9698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "0fd1f69a229d4b58851eabd27afa70dc"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "a6a0eaf76e8df0ab68eb55116e2debed"
server_key_iv: "8aa25d28f7ac97fe30a97b65ee07fa37"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.561043 9698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:57.575421 9698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:57.576468 9698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:57.576737 9698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:57.577582 9698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:57.579043 9698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:57.579116 9698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.579185 9698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:57.579214 9698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.591440 9698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45049
I20260925 16:27:57.591481 9825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45049 every 8 connection(s)
I20260925 16:27:57.592720 9698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:57.599309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9698
I20260925 16:27:57.599465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:57.599825 392 external_mini_cluster.cc:1428] Setting key 8c8ac0dd44a7da8142c17f3b4407c1c7
I20260925 16:27:57.602658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39767
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 16:27:57.606932 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:57.595287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35629 (local address 127.0.98.62:39767)
0925 16:27:57.595516 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:57.595520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:57.597004 (+ 1484us) server_negotiation.cc:407] Connection header received
0925 16:27:57.598931 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:57.598935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:57.599030 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:57.599121 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:57.600679 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.601522 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.603835 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.604137 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.604460 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:57.606198 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:57.606203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:57.606206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:57.606232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:57.606264 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:57.606276 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:57.606376 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:57.606600 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 16:27:57.606714 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 16:27:57.608201 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:57.595654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39767 (local address 127.0.98.1:35629)
0925 16:27:57.596731 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:57.596804 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:57.598581 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:57.599432 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:57.599445 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:57.599629 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:57.600356 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:57.600386 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.601875 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.601880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:57.603579 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:57.603606 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.604555 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.604561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:57.604818 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:57.605621 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:57.605623 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:57.605687 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:57.605847 (+ 160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:57.606414 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:57.606423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:57.606430 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:27:57.606573 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 16:27:57.606974 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":174,"threads_started":1}
I20260925 16:27:57.610260 9826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39767
I20260925 16:27:57.610632 9826 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:57.611451 9826 heartbeater.cc:511] Master 127.0.98.62:39767 requested a full tablet report, sending...
I20260925 16:27:57.613912 9638 ts_manager.cc:194] Registered new tserver with Master: 0fd1f69a229d4b58851eabd27afa70dc (127.0.98.1:45049)
I20260925 16:27:57.615571 9638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35629
W20260925 16:27:57.737630 9829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:57.738066 9829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:57.738112 9829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:57.744134 9829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:57.744228 9829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:57.744293 9829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:57.749855 9829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:57.751138 9829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:57.752684 9829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:57.765198 9837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:57.765180 9834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:57.765180 9835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:57.765264 9829 server_base.cc:1034] running on GCE node
I20260925 16:27:57.766217 9829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:57.767023 9829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:57.768263 9829 hybrid_clock.cc:648] HybridClock initialized: now 1790353677768229 us; error 69 us; skew 500 ppm
I20260925 16:27:57.770993 9829 webserver.cc:460] Webserver started at http://127.0.98.2:46125/ using document root <none> and password file <none>
I20260925 16:27:57.771868 9829 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:57.771962 9829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:57.772239 9829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:57.774791 9829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "80a76c6f110b41f19e9907fabca4a4d0"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "20b2283d9c8251f26965ec9ef47dbee9"
server_key_iv: "14028262319c177263c14e41e00ce0a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.775445 9829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80a76c6f110b41f19e9907fabca4a4d0"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "20b2283d9c8251f26965ec9ef47dbee9"
server_key_iv: "14028262319c177263c14e41e00ce0a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.780521 9829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 16:27:57.784909 9843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.786958 9829 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:57.787140 9829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "80a76c6f110b41f19e9907fabca4a4d0"
format_stamp: "Formatted at 2026-09-25 16:27:57 on dist-test-slave-31kt"
server_key: "20b2283d9c8251f26965ec9ef47dbee9"
server_key_iv: "14028262319c177263c14e41e00ce0a9"
server_key_version: "encryptionkey@0"
I20260925 16:27:57.787333 9829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:57.805090 9829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:57.806231 9829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:57.806464 9829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:57.807292 9829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:57.808835 9829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:57.808956 9829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.809031 9829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:57.809058 9829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:57.820528 9829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37721
I20260925 16:27:57.820570 9956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37721 every 8 connection(s)
I20260925 16:27:57.821882 9829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:57.823140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9829
I20260925 16:27:57.823246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:57.823603 392 external_mini_cluster.cc:1428] Setting key 0a980217b6a87bd8434fc6b4de5794c3
I20260925 16:27:57.826529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39767
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 16:27:57.833798 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:57.824509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39155 (local address 127.0.98.62:39767)
0925 16:27:57.824745 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:57.824750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:57.826411 (+ 1661us) server_negotiation.cc:407] Connection header received
0925 16:27:57.828370 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:57.828374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:57.828444 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:57.828522 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:57.829600 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.830193 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.831422 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.831649 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.831740 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:57.833108 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:57.833112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:57.833126 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:57.833143 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:57.833167 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:57.833178 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:57.833222 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:57.833486 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 16:27:57.833586 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 16:27:57.834797 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:57.825108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39767 (local address 127.0.98.2:39155)
0925 16:27:57.826201 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:57.826262 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:57.828092 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:57.828754 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:57.828763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:57.828867 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:57.829449 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:57.829463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.830321 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.830325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:57.831275 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:57.831287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:57.831813 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:57.831818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:57.831989 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:57.832748 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:57.832750 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:57.832812 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:57.832912 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:57.833284 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:57.833293 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:57.833299 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:27:57.833428 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 16:27:57.833787 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":206,"threads_started":1}
I20260925 16:27:57.836726 9957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39767
I20260925 16:27:57.837338 9957 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:57.839156 9957 heartbeater.cc:511] Master 127.0.98.62:39767 requested a full tablet report, sending...
I20260925 16:27:57.841053 9638 ts_manager.cc:194] Registered new tserver with Master: 80a76c6f110b41f19e9907fabca4a4d0 (127.0.98.2:37721)
I20260925 16:27:57.842006 9638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39155
W20260925 16:27:57.985219 9960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:57.985627 9960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:57.985673 9960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:57.990331 9960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:57.990459 9960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:57.990527 9960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:57.996294 9960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:57.998025 9960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:58.000150 9960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:58.013183 9968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.013199 9966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.013199 9965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:58.013559 9960 server_base.cc:1034] running on GCE node
I20260925 16:27:58.014182 9960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:58.015053 9960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:58.018326 9960 hybrid_clock.cc:648] HybridClock initialized: now 1790353678018296 us; error 52 us; skew 500 ppm
I20260925 16:27:58.021201 9960 webserver.cc:460] Webserver started at http://127.0.98.3:38291/ using document root <none> and password file <none>
I20260925 16:27:58.022109 9960 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:58.022202 9960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:58.022512 9960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:58.025601 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "6debacb41aa24f5f96cca52a96950936"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "c7d8334edaece0db976d8ceaafa5d033"
server_key_iv: "82518494402e5727bec3c53880107af8"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.026443 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6debacb41aa24f5f96cca52a96950936"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "c7d8334edaece0db976d8ceaafa5d033"
server_key_iv: "82518494402e5727bec3c53880107af8"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.031586 9960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:58.035647 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.037674 9960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 16:27:58.037962 9960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "6debacb41aa24f5f96cca52a96950936"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "c7d8334edaece0db976d8ceaafa5d033"
server_key_iv: "82518494402e5727bec3c53880107af8"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.038142 9960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:58.064553 9960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:58.065549 9960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:58.065811 9960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:58.066622 9960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:58.067925 9960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:58.067981 9960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.068029 9960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:58.068060 9960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.080021 9960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40197
I20260925 16:27:58.080042 10087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40197 every 8 connection(s)
I20260925 16:27:58.081312 9960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:58.087049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 9960
I20260925 16:27:58.087206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:58.087572 392 external_mini_cluster.cc:1428] Setting key edf21964f0c6caf1bd47a6c0858ffa19
I20260925 16:27:58.094131 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.083739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52375 (local address 127.0.98.62:39767)
0925 16:27:58.083975 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:58.083981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:58.085547 (+ 1566us) server_negotiation.cc:407] Connection header received
0925 16:27:58.087611 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:58.087616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:58.087698 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:58.087820 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:58.089242 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:58.089997 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:58.091509 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:27:58.091765 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:27:58.091938 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:58.093469 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:58.093473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:58.093476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:58.093496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:58.093525 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:58.093535 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:58.093610 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:58.093880 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 16:27:58.093966 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 16:27:58.095142 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.084209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39767 (local address 127.0.98.3:52375)
0925 16:27:58.085256 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:58.085320 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:58.087228 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:58.088163 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:58.088177 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:58.088322 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:58.089052 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:58.089072 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:58.090157 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:58.090161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:58.091356 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:27:58.091369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:27:58.092033 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:27:58.092037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:27:58.092249 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:27:58.093119 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:58.093120 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:58.093169 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:58.093262 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:58.093655 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:58.093664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:58.093669 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:27:58.093837 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 16:27:58.094140 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":171,"threads_started":1}
I20260925 16:27:58.096987 10088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39767
I20260925 16:27:58.097365 10088 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:58.098227 10088 heartbeater.cc:511] Master 127.0.98.62:39767 requested a full tablet report, sending...
I20260925 16:27:58.099835 9638 ts_manager.cc:194] Registered new tserver with Master: 6debacb41aa24f5f96cca52a96950936 (127.0.98.3:40197)
I20260925 16:27:58.100512 9638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52375
I20260925 16:27:58.102854 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:27:58.106597 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:58.105224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.0.98.62:39767)
0925 16:27:58.105438 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:58.105443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:58.105641 (+ 198us) server_negotiation.cc:407] Connection header received
0925 16:27:58.105744 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:58.105747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:58.105876 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:58.105989 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:58.106408 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54322: BlockingRecv error: recv got EOF from 127.0.0.1:54322 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 16:27:58.107255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9698
I20260925 16:27:58.114346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9829
I20260925 16:27:58.120649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9960
I20260925 16:27:58.126948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 9607
2026-09-25T16:27:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1138 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T16:27:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:27:58Z Disabled control of system clock
I20260925 16:27:58.160624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42265
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 16:27:58.310323 10099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:58.310750 10099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:58.310796 10099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:58.316313 10099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:27:58.316453 10099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:58.316469 10099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:58.316488 10099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:27:58.316504 10099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:27:58.323184 10099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:58.325007 10099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:58.327270 10099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:58.341403 10104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.341403 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.341478 10105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:58.341799 10099 server_base.cc:1034] running on GCE node
I20260925 16:27:58.342975 10099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:58.344795 10099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:58.346104 10099 hybrid_clock.cc:648] HybridClock initialized: now 1790353678346034 us; error 65 us; skew 500 ppm
I20260925 16:27:58.348953 10099 webserver.cc:460] Webserver started at http://127.0.98.62:46529/ using document root <none> and password file <none>
I20260925 16:27:58.349817 10099 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:58.349917 10099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:58.350159 10099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:58.352685 10099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "1a9d0e66f9c843ec989600c133aa5e68"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "b66f5d433e46e2802de611acdff62ec6"
server_key_iv: "61a239906699ea2db32f489968253ef2"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.353348 10099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a9d0e66f9c843ec989600c133aa5e68"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "b66f5d433e46e2802de611acdff62ec6"
server_key_iv: "61a239906699ea2db32f489968253ef2"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.362391 10099 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260925 16:27:58.367267 10113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.369704 10099 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:27:58.370096 10099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "1a9d0e66f9c843ec989600c133aa5e68"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "b66f5d433e46e2802de611acdff62ec6"
server_key_iv: "61a239906699ea2db32f489968253ef2"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.370302 10099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:58.377054 10099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:58.378192 10099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:58.378450 10099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:58.389325 10099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42265
I20260925 16:27:58.389353 10165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42265 every 8 connection(s)
I20260925 16:27:58.390676 10099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:27:58.395669 10166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:27:58.400878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10099
I20260925 16:27:58.401018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:27:58.401330 392 external_mini_cluster.cc:1428] Setting key 9c457769146cc8aa07cc3b86f5dc04ec
I20260925 16:27:58.405869 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68: Bootstrap starting.
I20260925 16:27:58.408916 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.403146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.0.98.62:42265)
0925 16:27:58.403986 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:58.404013 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:58.404057 (+ 44us) server_negotiation.cc:407] Connection header received
0925 16:27:58.404902 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:58.404925 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:58.405402 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:58.405822 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:58.407054 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:58.407062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:58.407081 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:58.407102 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:58.407190 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:58.407214 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:58.407385 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:58.407468 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 16:27:58.407834 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":174,"threads_started":1}
I20260925 16:27:58.409446 10166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68: Neither blocks nor log segments found. Creating new log.
I20260925 16:27:58.410476 10166 log.cc:824] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68: Log is configured to *not* fsync() on all Append() calls
I20260925 16:27:58.413386 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68: No bootstrap required, opened a new log
I20260925 16:27:58.418069 10166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9d0e66f9c843ec989600c133aa5e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42265 } }
I20260925 16:27:58.418483 10166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:27:58.418536 10166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9d0e66f9c843ec989600c133aa5e68, State: Initialized, Role: FOLLOWER
I20260925 16:27:58.419113 10166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9d0e66f9c843ec989600c133aa5e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42265 } }
I20260925 16:27:58.419278 10166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:27:58.419360 10166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:27:58.419474 10166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:27:58.421120 10166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9d0e66f9c843ec989600c133aa5e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42265 } }
I20260925 16:27:58.421845 10166 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a9d0e66f9c843ec989600c133aa5e68; no voters:
I20260925 16:27:58.422324 10166 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:27:58.422467 10171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:27:58.422691 10171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9d0e66f9c843ec989600c133aa5e68, State: Running, Role: LEADER
I20260925 16:27:58.423133 10171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9d0e66f9c843ec989600c133aa5e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42265 } }
I20260925 16:27:58.424389 10166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:27:58.425814 10173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9d0e66f9c843ec989600c133aa5e68. Latest consensus state: current_term: 1 leader_uuid: "1a9d0e66f9c843ec989600c133aa5e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9d0e66f9c843ec989600c133aa5e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42265 } } }
I20260925 16:27:58.425863 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a9d0e66f9c843ec989600c133aa5e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9d0e66f9c843ec989600c133aa5e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42265 } } }
I20260925 16:27:58.426061 10173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:58.426252 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68 [sys.catalog]: This master's current role is: LEADER
I20260925 16:27:58.426815 10180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:27:58.431723 10180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:27:58.441715 10180 catalog_manager.cc:1347] Generated new cluster ID: f41761dbd2114d5c992191c70b5ec4fb
I20260925 16:27:58.441931 10180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:27:58.471364 10180 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:27:58.472157 10180 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:27:58.484715 10180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a9d0e66f9c843ec989600c133aa5e68: Generated new TSK 0
I20260925 16:27:58.486057 10180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:27:58.496579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42265
--builtin_ntp_servers=127.0.98.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 16:27:58.645439 10190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:58.645844 10190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:58.645892 10190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:58.650270 10190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:58.650362 10190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:58.650424 10190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:27:58.655691 10190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:58.657112 10190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:58.658761 10190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:58.671129 10198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.671186 10195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.671133 10196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:58.671896 10190 server_base.cc:1034] running on GCE node
I20260925 16:27:58.672536 10190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:58.673544 10190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:58.677006 10190 hybrid_clock.cc:648] HybridClock initialized: now 1790353678676940 us; error 70 us; skew 500 ppm
I20260925 16:27:58.680124 10190 webserver.cc:460] Webserver started at http://127.0.98.1:36709/ using document root <none> and password file <none>
I20260925 16:27:58.680857 10190 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:58.680941 10190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:58.681160 10190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:58.683806 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "4bd1c00415bc4e2ca66c2a5f68f5fb47"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "d39d532539fe1a936c427d47754a188b"
server_key_iv: "23dd5c6c89da9a6937de89c5dc7b57ca"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.684544 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4bd1c00415bc4e2ca66c2a5f68f5fb47"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "d39d532539fe1a936c427d47754a188b"
server_key_iv: "23dd5c6c89da9a6937de89c5dc7b57ca"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.690236 10190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:27:58.693733 10204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.695721 10190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:27:58.695915 10190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "4bd1c00415bc4e2ca66c2a5f68f5fb47"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "d39d532539fe1a936c427d47754a188b"
server_key_iv: "23dd5c6c89da9a6937de89c5dc7b57ca"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.696058 10190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:58.701480 10190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:58.702296 10190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:58.702507 10190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:58.703186 10190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:58.704319 10190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:58.704375 10190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.704434 10190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:58.704463 10190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.716922 10190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34063
I20260925 16:27:58.717038 10317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34063 every 8 connection(s)
I20260925 16:27:58.718489 10190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:27:58.727443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10190
I20260925 16:27:58.727612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:27:58.727614 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.721311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49909 (local address 127.0.98.62:42265)
0925 16:27:58.721596 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:58.721601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:58.723166 (+ 1565us) server_negotiation.cc:407] Connection header received
0925 16:27:58.724962 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:58.724966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:58.725026 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:58.725140 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:58.726922 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:58.726927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:58.726929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:58.726947 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:58.726978 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:58.726989 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:58.727101 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:58.727393 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 16:27:58.727467 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260925 16:27:58.728022 392 external_mini_cluster.cc:1428] Setting key f9b7790f13d430b94668576d5f6032a1
I20260925 16:27:58.729202 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.721816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42265 (local address 127.0.98.1:49909)
0925 16:27:58.722884 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:58.722997 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:58.724568 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:58.725419 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:58.725430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:58.725573 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:58.726482 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:58.726484 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:58.726543 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:58.726652 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:58.727109 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:58.727122 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:58.727131 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 16:27:58.727439 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 16:27:58.727844 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":155,"threads_started":1}
I20260925 16:27:58.731096 10318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42265
I20260925 16:27:58.731349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42265
--builtin_ntp_servers=127.0.98.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 16:27:58.731544 10318 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:58.732277 10318 heartbeater.cc:511] Master 127.0.98.62:42265 requested a full tablet report, sending...
I20260925 16:27:58.735284 10130 ts_manager.cc:194] Registered new tserver with Master: 4bd1c00415bc4e2ca66c2a5f68f5fb47 (127.0.98.1:34063)
W20260925 16:27:58.880887 10321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:58.881357 10321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:58.881412 10321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:58.887485 10321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:58.887636 10321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:58.887722 10321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:27:58.897148 10321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:58.898991 10321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:58.900887 10321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:58.914448 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:58.914335 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:58.914674 10321 server_base.cc:1034] running on GCE node
W20260925 16:27:58.914362 10326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:58.915546 10321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:58.916525 10321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:58.917881 10321 hybrid_clock.cc:648] HybridClock initialized: now 1790353678917825 us; error 68 us; skew 500 ppm
I20260925 16:27:58.920787 10321 webserver.cc:460] Webserver started at http://127.0.98.2:39967/ using document root <none> and password file <none>
I20260925 16:27:58.921518 10321 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:58.921598 10321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:58.921886 10321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:58.925333 10321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "f2a196820bca437994951e849daa8dbb"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "01ca15375281dc02832f866691a6bb46"
server_key_iv: "a474781875dcfc3000bfaf7cede5ecaa"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.926880 10321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f2a196820bca437994951e849daa8dbb"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "01ca15375281dc02832f866691a6bb46"
server_key_iv: "a474781875dcfc3000bfaf7cede5ecaa"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.933698 10321 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260925 16:27:58.937729 10335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.939950 10321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:58.940140 10321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "f2a196820bca437994951e849daa8dbb"
format_stamp: "Formatted at 2026-09-25 16:27:58 on dist-test-slave-31kt"
server_key: "01ca15375281dc02832f866691a6bb46"
server_key_iv: "a474781875dcfc3000bfaf7cede5ecaa"
server_key_version: "encryptionkey@0"
I20260925 16:27:58.940352 10321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:58.946719 10321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:58.947563 10321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:58.947765 10321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:58.948442 10321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:58.949925 10321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:58.949991 10321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.950054 10321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:58.950093 10321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:58.962177 10321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34943
I20260925 16:27:58.962225 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34943 every 8 connection(s)
I20260925 16:27:58.963294 10321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:27:58.972069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10321
I20260925 16:27:58.972226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:27:58.972360 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.966072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49819 (local address 127.0.98.62:42265)
0925 16:27:58.966290 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:58.966296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:58.968102 (+ 1806us) server_negotiation.cc:407] Connection header received
0925 16:27:58.969831 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:58.969836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:58.969904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:58.970004 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:58.971683 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:58.971690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:58.971694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:58.971719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:58.971765 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:58.971781 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:58.971950 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:58.972102 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 16:27:58.972165 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 16:27:58.972572 392 external_mini_cluster.cc:1428] Setting key 2be03f1d78abf628a905ac4cbb8c916c
I20260925 16:27:58.974221 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:58.966690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42265 (local address 127.0.98.2:49819)
0925 16:27:58.967845 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:58.967906 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:58.969444 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:58.970295 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:58.970306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:58.970452 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:58.971203 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:58.971205 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:58.971262 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:58.971376 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:58.971936 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:58.971944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:58.971949 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:27:58.972140 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 16:27:58.972556 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":224,"threads_started":1}
I20260925 16:27:58.975284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42265
--builtin_ntp_servers=127.0.98.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 16:27:58.976099 10449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42265
I20260925 16:27:58.976464 10449 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:58.977173 10449 heartbeater.cc:511] Master 127.0.98.62:42265 requested a full tablet report, sending...
I20260925 16:27:58.979044 10130 ts_manager.cc:194] Registered new tserver with Master: f2a196820bca437994951e849daa8dbb (127.0.98.2:34943)
W20260925 16:27:59.119364 10452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:27:59.119750 10452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:27:59.119812 10452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:27:59.124601 10452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:27:59.124716 10452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:27:59.124805 10452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:27:59.132295 10452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:27:59.134184 10452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:27:59.136129 10452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:27:59.148641 10460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:59.148523 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:27:59.148523 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:27:59.150281 10452 server_base.cc:1034] running on GCE node
I20260925 16:27:59.151137 10452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:27:59.152261 10452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:27:59.153523 10452 hybrid_clock.cc:648] HybridClock initialized: now 1790353679153477 us; error 67 us; skew 500 ppm
I20260925 16:27:59.156875 10452 webserver.cc:460] Webserver started at http://127.0.98.3:38103/ using document root <none> and password file <none>
I20260925 16:27:59.157863 10452 fs_manager.cc:359] Metadata directory not provided
I20260925 16:27:59.157986 10452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:27:59.158300 10452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:27:59.161412 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "e14cc6451d3e46ccbede1444fa2a16dd"
format_stamp: "Formatted at 2026-09-25 16:27:59 on dist-test-slave-31kt"
server_key: "76c73ca1b377a40ce79ab731c3cbb1e7"
server_key_iv: "6577e0f8f1b6774a27a43b8929ed4c51"
server_key_version: "encryptionkey@0"
I20260925 16:27:59.162407 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e14cc6451d3e46ccbede1444fa2a16dd"
format_stamp: "Formatted at 2026-09-25 16:27:59 on dist-test-slave-31kt"
server_key: "76c73ca1b377a40ce79ab731c3cbb1e7"
server_key_iv: "6577e0f8f1b6774a27a43b8929ed4c51"
server_key_version: "encryptionkey@0"
I20260925 16:27:59.168397 10452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260925 16:27:59.172472 10466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:59.174988 10452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:27:59.175473 10452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "e14cc6451d3e46ccbede1444fa2a16dd"
format_stamp: "Formatted at 2026-09-25 16:27:59 on dist-test-slave-31kt"
server_key: "76c73ca1b377a40ce79ab731c3cbb1e7"
server_key_iv: "6577e0f8f1b6774a27a43b8929ed4c51"
server_key_version: "encryptionkey@0"
I20260925 16:27:59.175710 10452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:27:59.182605 10452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:27:59.183604 10452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:27:59.183836 10452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:27:59.184697 10452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:27:59.186134 10452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:27:59.186193 10452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:59.186264 10452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:27:59.186290 10452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:27:59.198658 10452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39143
I20260925 16:27:59.198734 10579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39143 every 8 connection(s)
I20260925 16:27:59.200073 10452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:27:59.205093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10452
I20260925 16:27:59.205235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:27:59.205628 392 external_mini_cluster.cc:1428] Setting key 5ced168b995d8e26cdb09d1be9e19bcd
I20260925 16:27:59.209607 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:59.202743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58517 (local address 127.0.98.62:42265)
0925 16:27:59.202956 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:59.202961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:59.204816 (+ 1855us) server_negotiation.cc:407] Connection header received
0925 16:27:59.206971 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:59.206975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:59.207041 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:59.207150 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:59.209041 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:27:59.209045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:27:59.209048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:27:59.209063 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:27:59.209095 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:27:59.209103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:27:59.209202 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 16:27:59.209397 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 16:27:59.209454 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 16:27:59.211122 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:27:59.203364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42265 (local address 127.0.98.3:58517)
0925 16:27:59.204512 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0925 16:27:59.204583 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:27:59.206463 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:27:59.207684 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:27:59.207697 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:27:59.207836 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:27:59.208683 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:27:59.208685 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:27:59.208756 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:27:59.208858 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:27:59.209225 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:27:59.209234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:27:59.209239 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:27:59.209428 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:27:59.209729 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":161,"threads_started":1}
I20260925 16:27:59.212862 10580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42265
I20260925 16:27:59.213279 10580 heartbeater.cc:464] Registering TS with master...
I20260925 16:27:59.214095 10580 heartbeater.cc:511] Master 127.0.98.62:42265 requested a full tablet report, sending...
I20260925 16:27:59.215758 10130 ts_manager.cc:194] Registered new tserver with Master: e14cc6451d3e46ccbede1444fa2a16dd (127.0.98.3:39143)
I20260925 16:27:59.222004 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 16:27:59.225911 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:27:59.224657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.0.98.62:42265)
0925 16:27:59.224902 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 16:27:59.224907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:27:59.225044 (+ 137us) server_negotiation.cc:407] Connection header received
0925 16:27:59.225249 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:27:59.225253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:27:59.225307 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:27:59.225391 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:27:59.225676 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54852: BlockingRecv error: recv got EOF from 127.0.0.1:54852 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 16:27:59.226132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10190
I20260925 16:27:59.233008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10321
I20260925 16:27:59.239502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10452
I20260925 16:27:59.245244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10099
2026-09-25T16:27:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1118 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:27:59 dist-test-slave-31kt krb5kdc[10590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:27:59 dist-test-slave-31kt krb5kdc[10590](info): set up 2 sockets
Sep 25 16:27:59 dist-test-slave-31kt krb5kdc[10590](info): commencing operation
krb5kdc: starting...
W20260925 16:28:00.246965 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:28:00 dist-test-slave-31kt krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:28:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:28:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:28:00.394105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:28:00.529678 10603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:00.530054 10603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:00.530112 10603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:00.534461 10603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:28:00.534552 10603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:00.534579 10603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:00.534600 10603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:28:00.534621 10603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:28:00.543169 10603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:00.545681 10603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:00.547128 10603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:00.559831 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:00.559818 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:00.559818 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:00.560413 10603 server_base.cc:1034] running on GCE node
I20260925 16:28:00.561262 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:00.563241 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:00.564815 10603 hybrid_clock.cc:648] HybridClock initialized: now 1790353680564765 us; error 62 us; skew 500 ppm
Sep 25 16:28:00 dist-test-slave-31kt krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:00.577409 10603 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:28:00.578855 10603 webserver.cc:460] Webserver started at http://127.0.98.62:32793/ using document root <none> and password file <none>
I20260925 16:28:00.579797 10603 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:00.580089 10603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:00.580474 10603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:28:00.583777 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "d40c45051294441d979dbc78b97a067e"
format_stamp: "Formatted at 2026-09-25 16:28:00 on dist-test-slave-31kt"
server_key: "c24d1ec2605907cb79205593f207868d"
server_key_iv: "85afb73e45690515433a8505aa819753"
server_key_version: "encryptionkey@0"
I20260925 16:28:00.584884 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d40c45051294441d979dbc78b97a067e"
format_stamp: "Formatted at 2026-09-25 16:28:00 on dist-test-slave-31kt"
server_key: "c24d1ec2605907cb79205593f207868d"
server_key_iv: "85afb73e45690515433a8505aa819753"
server_key_version: "encryptionkey@0"
I20260925 16:28:00.591094 10603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 16:28:00.594942 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:00.597705 10603 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260925 16:28:00.598018 10603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "d40c45051294441d979dbc78b97a067e"
format_stamp: "Formatted at 2026-09-25 16:28:00 on dist-test-slave-31kt"
server_key: "c24d1ec2605907cb79205593f207868d"
server_key_iv: "85afb73e45690515433a8505aa819753"
server_key_version: "encryptionkey@0"
I20260925 16:28:00.598178 10603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:00.620826 10603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:00.625341 10603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:00.625739 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:00.635262 10603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35353
I20260925 16:28:00.635633 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35353 every 8 connection(s)
I20260925 16:28:00.636466 10603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:28:00.640898 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:00.644652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10603
I20260925 16:28:00.644817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:28:00.645278 392 external_mini_cluster.cc:1428] Setting key e86734e84a732de1530a7fb9d82daca7
I20260925 16:28:00.649715 10671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: Bootstrap starting.
I20260925 16:28:00.652697 10671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:00.653683 10671 log.cc:824] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:00.656709 10671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: No bootstrap required, opened a new log
Sep 25 16:28:00 dist-test-slave-31kt krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:28:00.660382 10671 raft_consensus.cc:348] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c45051294441d979dbc78b97a067e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260925 16:28:00.660676 10671 raft_consensus.cc:374] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:00.660728 10671 raft_consensus.cc:729] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40c45051294441d979dbc78b97a067e, State: Initialized, Role: FOLLOWER
I20260925 16:28:00.661321 10671 consensus_queue.cc:260] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c45051294441d979dbc78b97a067e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260925 16:28:00.661506 10671 raft_consensus.cc:388] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:28:00.661578 10671 raft_consensus.cc:482] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:28:00.661708 10671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:00.663456 10671 raft_consensus.cc:504] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c45051294441d979dbc78b97a067e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260925 16:28:00.663956 10671 leader_election.cc:302] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d40c45051294441d979dbc78b97a067e; no voters:
I20260925 16:28:00.664372 10671 leader_election.cc:288] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:28:00.664554 10676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:28:00.664975 10676 raft_consensus.cc:686] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [term 1 LEADER]: Becoming Leader. State: Replica: d40c45051294441d979dbc78b97a067e, State: Running, Role: LEADER
I20260925 16:28:00.665514 10676 consensus_queue.cc:237] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c45051294441d979dbc78b97a067e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260925 16:28:00.665865 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:00.647322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.98.62:35353)
0925 16:28:00.647917 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:00.647937 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:00.647988 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:28:00.648881 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:00.648912 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:00.649419 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:00.649935 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:00.650923 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:00.652176 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:00.653492 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:00.654331 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:00.654552 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:00.657871 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:00.657880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:00.657904 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:00.657947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:00.660669 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:00.661385 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:00.661392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:00.661399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:00.661560 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:00.661952 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:00.661956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:00.661958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:00.662540 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:00.662773 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:00.663929 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0925 16:28:00.664408 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":178,"threads_started":1}
I20260925 16:28:00.665984 10671 sys_catalog.cc:564] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:28:00.667850 10678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d40c45051294441d979dbc78b97a067e. Latest consensus state: current_term: 1 leader_uuid: "d40c45051294441d979dbc78b97a067e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c45051294441d979dbc78b97a067e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } } }
I20260925 16:28:00.668063 10678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [sys.catalog]: This master's current role is: LEADER
I20260925 16:28:00.668529 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d40c45051294441d979dbc78b97a067e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c45051294441d979dbc78b97a067e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } } }
I20260925 16:28:00.669159 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e [sys.catalog]: This master's current role is: LEADER
W20260925 16:28:00.672724 10691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:28:00.672859 10691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:28:00.672982 10690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:28:00.674103 10690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:28:00.683212 10690 catalog_manager.cc:1347] Generated new cluster ID: c1e2b33d4aa64086b3ef0678abed1b19
I20260925 16:28:00.683353 10690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:28:00.707502 10690 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:28:00.709069 10690 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:28:00.718684 10690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: Generated new TSK 0
I20260925 16:28:00.719743 10690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:28:00.757655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35353
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:28:00.898507 10697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:00.898957 10697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:00.899010 10697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:00.903812 10697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:00.903929 10697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:00.904003 10697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:28:00.912118 10697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:00.913681 10697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:00.915071 10697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:00.927553 10703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:00.927551 10702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:00.927541 10705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:00.928048 10697 server_base.cc:1034] running on GCE node
I20260925 16:28:00.928647 10697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:00.929570 10697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:00.930795 10697 hybrid_clock.cc:648] HybridClock initialized: now 1790353680930765 us; error 62 us; skew 500 ppm
Sep 25 16:28:00 dist-test-slave-31kt krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:00.934489 10697 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:28:00.935827 10697 webserver.cc:460] Webserver started at http://127.0.98.1:34743/ using document root <none> and password file <none>
I20260925 16:28:00.936549 10697 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:00.936632 10697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:00.936880 10697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:28:00.939878 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa44cfa2801a4e65a866aa8fab126296"
format_stamp: "Formatted at 2026-09-25 16:28:00 on dist-test-slave-31kt"
server_key: "392ebceff4aa33c28789822cf3725ceb"
server_key_iv: "54232fd8dc0549be0810aa8adc366674"
server_key_version: "encryptionkey@0"
I20260925 16:28:00.940634 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa44cfa2801a4e65a866aa8fab126296"
format_stamp: "Formatted at 2026-09-25 16:28:00 on dist-test-slave-31kt"
server_key: "392ebceff4aa33c28789822cf3725ceb"
server_key_iv: "54232fd8dc0549be0810aa8adc366674"
server_key_version: "encryptionkey@0"
I20260925 16:28:00.946158 10697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 16:28:00.950088 10712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:00.952037 10697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:28:00.952244 10697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "fa44cfa2801a4e65a866aa8fab126296"
format_stamp: "Formatted at 2026-09-25 16:28:00 on dist-test-slave-31kt"
server_key: "392ebceff4aa33c28789822cf3725ceb"
server_key_iv: "54232fd8dc0549be0810aa8adc366674"
server_key_version: "encryptionkey@0"
I20260925 16:28:00.952410 10697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:00.974852 10697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:00.979229 10697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:00.979499 10697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:00.980386 10697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:28:00.981563 10697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:28:00.981621 10697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:00.981686 10697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:28:00.981711 10697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:00.993402 10697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35965
I20260925 16:28:00.993544 10825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35965 every 8 connection(s)
I20260925 16:28:00.994652 10697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:28:00.997089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10697
I20260925 16:28:00.997238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:28:00.997602 392 external_mini_cluster.cc:1428] Setting key 130496c5de8019e8ada3a806d95876c1
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 16:28:01 dist-test-slave-31kt krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:28:01.020529 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:00.997505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48593 (local address 127.0.98.62:35353)
0925 16:28:00.997710 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:00.997716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:00.999473 (+ 1757us) server_negotiation.cc:407] Connection header received
0925 16:28:01.001591 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.001595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.001665 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.001805 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:01.004020 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.004890 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.006787 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.007208 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.007381 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.015063 (+ 7682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:01.015068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:01.015070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:01.015115 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:01.017206 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.018254 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.018260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.018262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.018405 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.018907 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.018910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.018912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.019203 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:01.019354 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.020140 (+ 786us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.020316 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:28:01.021874 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:00.998404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35353 (local address 127.0.98.1:48593)
0925 16:28:00.999200 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:00.999295 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.001146 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.002138 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.002150 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.002898 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:28:01.003710 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.003736 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.005135 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.005140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.006538 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.006562 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.007491 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.007495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.007742 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.008677 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:28:01.008701 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:28:01.014528 (+ 5827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:28:01.017467 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:01.017477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:01.017504 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:01.018057 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:01.018525 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:01.018527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:01.018529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:01.018767 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:01.019383 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:28:01.019391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:28:01.019804 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.020030 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.020476 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":159,"threads_started":1}
I20260925 16:28:01.023931 10826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35353
I20260925 16:28:01.024255 10826 heartbeater.cc:464] Registering TS with master...
I20260925 16:28:01.024936 10826 heartbeater.cc:511] Master 127.0.98.62:35353 requested a full tablet report, sending...
I20260925 16:28:01.027076 10633 ts_manager.cc:194] Registered new tserver with Master: fa44cfa2801a4e65a866aa8fab126296 (127.0.98.1:35965)
I20260925 16:28:01.028393 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:28:01.034729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35353
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:28:01.168517 10831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:01.169134 10831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:01.169194 10831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:01.173712 10831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:01.173870 10831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:01.173961 10831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:28:01.182030 10831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:01.183785 10831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:01.185365 10831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:01.199149 10836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:01.199164 10839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:01.199149 10837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:01.199668 10831 server_base.cc:1034] running on GCE node
I20260925 16:28:01.200596 10831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:01.201817 10831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:01.203116 10831 hybrid_clock.cc:648] HybridClock initialized: now 1790353681203102 us; error 51 us; skew 500 ppm
Sep 25 16:28:01 dist-test-slave-31kt krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:01.207063 10831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:28:01.208308 10831 webserver.cc:460] Webserver started at http://127.0.98.2:43611/ using document root <none> and password file <none>
I20260925 16:28:01.209107 10831 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:01.209200 10831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:01.209427 10831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:28:01.211850 10831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "76571d16dc574af5978a907d75ffb63b"
format_stamp: "Formatted at 2026-09-25 16:28:01 on dist-test-slave-31kt"
server_key: "83711bbdeff35dbab2640a8e87593c0d"
server_key_iv: "790aa7551bd5cfcffcb0fe770e7dd674"
server_key_version: "encryptionkey@0"
I20260925 16:28:01.212487 10831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76571d16dc574af5978a907d75ffb63b"
format_stamp: "Formatted at 2026-09-25 16:28:01 on dist-test-slave-31kt"
server_key: "83711bbdeff35dbab2640a8e87593c0d"
server_key_iv: "790aa7551bd5cfcffcb0fe770e7dd674"
server_key_version: "encryptionkey@0"
I20260925 16:28:01.217556 10831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:28:01.221660 10846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:01.224102 10831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:28:01.224313 10831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "76571d16dc574af5978a907d75ffb63b"
format_stamp: "Formatted at 2026-09-25 16:28:01 on dist-test-slave-31kt"
server_key: "83711bbdeff35dbab2640a8e87593c0d"
server_key_iv: "790aa7551bd5cfcffcb0fe770e7dd674"
server_key_version: "encryptionkey@0"
I20260925 16:28:01.224614 10831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:01.242005 10831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:01.245885 10831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:01.246097 10831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:01.246764 10831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:28:01.247887 10831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:28:01.247946 10831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:01.248005 10831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:28:01.248034 10831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:01.259138 10831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43495
I20260925 16:28:01.259236 10959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43495 every 8 connection(s)
I20260925 16:28:01.260238 10831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:28:01.264389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10831
I20260925 16:28:01.264531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:28:01.264868 392 external_mini_cluster.cc:1428] Setting key a95b3197c5d97790984e20a4ad731627
Sep 25 16:28:01 dist-test-slave-31kt krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:28:01.280099 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.262851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47011 (local address 127.0.98.62:35353)
0925 16:28:01.263050 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.263054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.264238 (+ 1184us) server_negotiation.cc:407] Connection header received
0925 16:28:01.266042 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.266045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.266099 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.266177 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:01.268551 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.269312 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.271303 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.271595 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.271709 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.275526 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:01.275531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:01.275534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:01.275573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:01.277221 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.278025 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.278029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.278031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.278123 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.278602 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.278605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.278606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.278868 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:01.278960 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.279699 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.279892 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 16:28:01.281373 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.263344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35353 (local address 127.0.98.2:47011)
0925 16:28:01.264004 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.264063 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.265719 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.266517 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.266529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.267108 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:28:01.268238 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.268270 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.269516 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.269520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.271084 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.271104 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.271834 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.271839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.272098 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.272985 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:28:01.273014 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:28:01.275205 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:28:01.277408 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:01.277418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:01.277445 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:01.277897 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:01.278223 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:01.278227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:01.278229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:01.278462 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:01.278973 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:28:01.278984 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:28:01.279407 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.279630 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.280117 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":161,"threads_started":1}
I20260925 16:28:01.283144 10960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35353
I20260925 16:28:01.283517 10960 heartbeater.cc:464] Registering TS with master...
I20260925 16:28:01.284492 10960 heartbeater.cc:511] Master 127.0.98.62:35353 requested a full tablet report, sending...
I20260925 16:28:01.286391 10633 ts_manager.cc:194] Registered new tserver with Master: 76571d16dc574af5978a907d75ffb63b (127.0.98.2:43495)
I20260925 16:28:01.287055 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:28:01.301316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35353
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:28:01.443862 10965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:01.444353 10965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:01.444429 10965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:01.449626 10965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:01.449740 10965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:01.449872 10965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:28:01.457620 10965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:01.459844 10965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:01.461987 10965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:01.478456 10970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:01.478328 10973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:01.478328 10971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:01.478933 10965 server_base.cc:1034] running on GCE node
I20260925 16:28:01.479630 10965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:01.480760 10965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:01.484177 10965 hybrid_clock.cc:648] HybridClock initialized: now 1790353681484084 us; error 97 us; skew 500 ppm
Sep 25 16:28:01 dist-test-slave-31kt krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:01.488838 10965 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:28:01.490582 10965 webserver.cc:460] Webserver started at http://127.0.98.3:46785/ using document root <none> and password file <none>
I20260925 16:28:01.491484 10965 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:01.491591 10965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:01.491866 10965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:28:01.495102 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c"
format_stamp: "Formatted at 2026-09-25 16:28:01 on dist-test-slave-31kt"
server_key: "87c1532af9f1f416806feec06023defe"
server_key_iv: "b35b30d02571679ad39c3daa4e018672"
server_key_version: "encryptionkey@0"
I20260925 16:28:01.496002 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c"
format_stamp: "Formatted at 2026-09-25 16:28:01 on dist-test-slave-31kt"
server_key: "87c1532af9f1f416806feec06023defe"
server_key_iv: "b35b30d02571679ad39c3daa4e018672"
server_key_version: "encryptionkey@0"
I20260925 16:28:01.501672 10965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:28:01.505899 10980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:01.508020 10965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:28:01.508246 10965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c"
format_stamp: "Formatted at 2026-09-25 16:28:01 on dist-test-slave-31kt"
server_key: "87c1532af9f1f416806feec06023defe"
server_key_iv: "b35b30d02571679ad39c3daa4e018672"
server_key_version: "encryptionkey@0"
I20260925 16:28:01.508424 10965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:01.531428 10965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:01.535534 10965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:01.535741 10965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:01.536473 10965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:28:01.537616 10965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:28:01.537688 10965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:01.537743 10965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:28:01.537799 10965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 16:28:01.549823 10965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37403
I20260925 16:28:01.549854 11093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37403 every 8 connection(s)
I20260925 16:28:01.551146 10965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:28:01.551609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 10965
I20260925 16:28:01.551772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:28:01.552166 392 external_mini_cluster.cc:1428] Setting key adeb7900d3dbde3caa45c4ea4a09f4d4
Sep 25 16:28:01 dist-test-slave-31kt krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:28:01.571612 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.554112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54029 (local address 127.0.98.62:35353)
0925 16:28:01.554328 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.554334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.555850 (+ 1516us) server_negotiation.cc:407] Connection header received
0925 16:28:01.557434 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.557437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.557514 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.557644 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:01.559770 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.560442 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.562244 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.562553 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.562739 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.566477 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:01.566483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:01.566487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:01.566546 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:01.568654 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.569389 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.569393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.569394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.569510 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.570121 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.570125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.570126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.570384 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:01.570482 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.571209 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.571386 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:28:01.572497 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.554878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35353 (local address 127.0.98.3:54029)
0925 16:28:01.555656 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.555726 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.557137 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.558123 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.558133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.558782 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:28:01.559529 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.559550 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.560837 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.560842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.562017 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.562037 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.562908 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.562913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.563203 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.564162 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:28:01.564188 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:28:01.566196 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:28:01.568820 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:01.568827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:01.568848 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:01.569246 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:01.569677 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:01.569680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:01.569682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:01.569968 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:01.570538 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:28:01.570545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:28:01.570889 (+ 344us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.571088 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.571464 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":201,"threads_started":1}
I20260925 16:28:01.574303 11094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35353
I20260925 16:28:01.574658 11094 heartbeater.cc:464] Registering TS with master...
I20260925 16:28:01.575295 11094 heartbeater.cc:511] Master 127.0.98.62:35353 requested a full tablet report, sending...
I20260925 16:28:01.576938 10633 ts_manager.cc:194] Registered new tserver with Master: f6a63ca96ba6477b866e1cf71ee7ee9c (127.0.98.3:37403)
I20260925 16:28:01.577852 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54029
I20260925 16:28:01.581439 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:28:01.594832 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.584749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.98.62:35353)
0925 16:28:01.584989 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.584995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.585123 (+ 128us) server_negotiation.cc:407] Connection header received
0925 16:28:01.585316 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.585320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.585373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.585449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:01.586403 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.586991 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.587930 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.588164 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.588261 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.590190 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:01.590196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:01.590199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:01.590246 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:01.592244 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.593030 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.593034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.593035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.593152 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:01.593527 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:01.593530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:01.593532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:01.593915 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:01.594058 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.594462 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.594641 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 16:28:01.599550 10633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:28:01.604031 10633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:28:01.625730 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.619487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.0.98.3:37403)
0925 16:28:01.620029 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.620042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.620067 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:28:01.620160 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.620166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.620409 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.620553 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:01.620826 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.621720 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.623687 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.624734 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.624859 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.625107 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.625373 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.625517 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":203,"threads_started":1}
I20260925 16:28:01.625723 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.618780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37403 (local address 127.0.0.1:56458)
0925 16:28:01.619323 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.619336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.619427 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.620542 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.620545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.620562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:01.620716 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.620724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.621891 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.621894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.623487 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.623497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.624883 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.624886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.625131 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.625184 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.625313 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.625503 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":162,"thread_start_us":130,"threads_started":1}
I20260925 16:28:01.626259 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.618357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35965 (local address 127.0.0.1:39106)
0925 16:28:01.618814 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.618893 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.619058 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.619740 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.619747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.619775 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:01.620006 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.620014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.621805 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.621808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.623811 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.623822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.624941 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.624945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.625859 (+ 914us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.625889 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.625975 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.626074 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":872,"thread_start_us":100,"threads_started":1}
I20260925 16:28:01.627281 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.618544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.0.98.1:35965)
0925 16:28:01.619076 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.619085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.619117 (+ 32us) server_negotiation.cc:407] Connection header received
0925 16:28:01.619218 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.619224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.619532 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.620200 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:01.620380 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.621638 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.623990 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.624791 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.625112 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.625579 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.626877 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.627074 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":143,"threads_started":1}
I20260925 16:28:01.627367 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.619820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.98.2:43495)
0925 16:28:01.620586 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.620598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.620621 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:28:01.620702 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.620707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.620892 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.621049 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:01.621257 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.622490 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.624492 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.625901 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.626006 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.626174 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.627069 (+ 895us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.627210 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":146,"threads_started":1}
I20260925 16:28:01.627712 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.619342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43495 (local address 127.0.0.1:50186)
0925 16:28:01.620232 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.620246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.620339 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.621041 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.621044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.621063 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:01.621152 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.621158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.622688 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.622691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.624328 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.624338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.626245 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.626249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.626342 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.626365 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.626436 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.626517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":106,"threads_started":1}
I20260925 16:28:01.631810 11028 tablet_service.cc:1467] Processing CreateTablet for tablet 32714952d99f4e68ae205e78f42b2f7c (DEFAULT_TABLE table=test-table [id=d22e85c695ec4a40acb1f30d5aff953e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:28:01.632860 10894 tablet_service.cc:1467] Processing CreateTablet for tablet 32714952d99f4e68ae205e78f42b2f7c (DEFAULT_TABLE table=test-table [id=d22e85c695ec4a40acb1f30d5aff953e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:28:01.633733 11028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32714952d99f4e68ae205e78f42b2f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:01.634105 10894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32714952d99f4e68ae205e78f42b2f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:01.633730 10760 tablet_service.cc:1467] Processing CreateTablet for tablet 32714952d99f4e68ae205e78f42b2f7c (DEFAULT_TABLE table=test-table [id=d22e85c695ec4a40acb1f30d5aff953e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:28:01.635213 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32714952d99f4e68ae205e78f42b2f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:01.648742 11111 tablet_bootstrap.cc:492] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Bootstrap starting.
I20260925 16:28:01.649470 11112 tablet_bootstrap.cc:492] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Bootstrap starting.
I20260925 16:28:01.652170 11111 tablet_bootstrap.cc:654] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:01.653002 11112 tablet_bootstrap.cc:654] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:01.654089 11111 log.cc:824] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:01.654279 11113 tablet_bootstrap.cc:492] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: Bootstrap starting.
I20260925 16:28:01.655896 11112 log.cc:824] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:01.657261 11113 tablet_bootstrap.cc:654] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:01.657598 11111 tablet_bootstrap.cc:492] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: No bootstrap required, opened a new log
I20260925 16:28:01.657912 11111 ts_tablet_manager.cc:1397] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20260925 16:28:01.658779 11113 log.cc:824] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:01.659464 11112 tablet_bootstrap.cc:492] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: No bootstrap required, opened a new log
I20260925 16:28:01.659731 11112 ts_tablet_manager.cc:1397] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 16:28:01.661396 11113 tablet_bootstrap.cc:492] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: No bootstrap required, opened a new log
I20260925 16:28:01.661631 11113 ts_tablet_manager.cc:1397] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 16:28:01.662395 11111 raft_consensus.cc:348] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.662827 11111 raft_consensus.cc:374] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:01.662882 11111 raft_consensus.cc:729] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a63ca96ba6477b866e1cf71ee7ee9c, State: Initialized, Role: FOLLOWER
I20260925 16:28:01.663522 11111 consensus_queue.cc:260] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.664331 11112 raft_consensus.cc:348] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.664788 11112 raft_consensus.cc:374] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:01.664880 11112 raft_consensus.cc:729] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76571d16dc574af5978a907d75ffb63b, State: Initialized, Role: FOLLOWER
I20260925 16:28:01.665076 11111 ts_tablet_manager.cc:1428] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260925 16:28:01.665585 11094 heartbeater.cc:503] Master 127.0.98.62:35353 was elected leader, sending a full tablet report...
I20260925 16:28:01.665684 11112 consensus_queue.cc:260] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.666669 10960 heartbeater.cc:503] Master 127.0.98.62:35353 was elected leader, sending a full tablet report...
I20260925 16:28:01.667125 11112 ts_tablet_manager.cc:1428] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:28:01.668483 11113 raft_consensus.cc:348] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.668812 11113 raft_consensus.cc:374] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:01.668883 11113 raft_consensus.cc:729] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa44cfa2801a4e65a866aa8fab126296, State: Initialized, Role: FOLLOWER
I20260925 16:28:01.669423 11113 consensus_queue.cc:260] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.670626 10826 heartbeater.cc:503] Master 127.0.98.62:35353 was elected leader, sending a full tablet report...
I20260925 16:28:01.672531 11113 ts_tablet_manager.cc:1428] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
W20260925 16:28:01.746522 10827 tablet.cc:2369] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:28:01.761783 10961 tablet.cc:2369] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:28:01.802981 11095 tablet.cc:2369] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:28:01.965991 11119 raft_consensus.cc:482] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:28:01.966410 11119 raft_consensus.cc:504] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.968164 11119 leader_election.cc:288] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76571d16dc574af5978a907d75ffb63b (127.0.98.2:43495), f6a63ca96ba6477b866e1cf71ee7ee9c (127.0.98.3:37403)
I20260925 16:28:01.975724 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.968795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51385 (local address 127.0.98.2:43495)
0925 16:28:01.968910 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.968915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.969323 (+ 408us) server_negotiation.cc:407] Connection header received
0925 16:28:01.969556 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.969561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.969640 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.969793 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:01.970610 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.971473 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.973825 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.974587 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.974772 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.974874 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.975398 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.975529 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:28:01.975724 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.968697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40107 (local address 127.0.98.3:37403)
0925 16:28:01.968872 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:01.968876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:01.969322 (+ 446us) server_negotiation.cc:407] Connection header received
0925 16:28:01.969556 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:01.969561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:01.969638 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:01.969825 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:01.970578 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.971482 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.973825 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.974587 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.974772 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.974874 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:01.975398 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 16:28:01.975524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:28:01.975757 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.968637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43495 (local address 127.0.98.1:51385)
0925 16:28:01.969126 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.969145 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.969366 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.969882 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.969886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.969925 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:01.970392 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.970407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.971699 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.971705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.973466 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.973492 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.974875 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.974881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.975149 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.975195 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.975325 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.975494 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":398,"spinlock_wait_cycles":33152,"thread_start_us":165,"threads_started":1}
I20260925 16:28:01.975757 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:01.968674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37403 (local address 127.0.98.1:40107)
0925 16:28:01.969192 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:01.969204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:01.969366 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:01.969882 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:01.969886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:01.969925 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:01.970400 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.970407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.971699 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.971705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.973466 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:01.973492 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:01.974875 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:01.974888 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:01.975141 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:01.975195 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 16:28:01.975325 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 16:28:01.975494 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":363,"spinlock_wait_cycles":34816,"thread_start_us":146,"threads_started":1}
I20260925 16:28:01.976851 10914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32714952d99f4e68ae205e78f42b2f7c" candidate_uuid: "fa44cfa2801a4e65a866aa8fab126296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76571d16dc574af5978a907d75ffb63b" is_pre_election: true
I20260925 16:28:01.977221 10914 raft_consensus.cc:2454] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa44cfa2801a4e65a866aa8fab126296 in term 0.
I20260925 16:28:01.977231 11048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32714952d99f4e68ae205e78f42b2f7c" candidate_uuid: "fa44cfa2801a4e65a866aa8fab126296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" is_pre_election: true
I20260925 16:28:01.977665 11048 raft_consensus.cc:2454] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa44cfa2801a4e65a866aa8fab126296 in term 0.
I20260925 16:28:01.978199 10716 leader_election.cc:302] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76571d16dc574af5978a907d75ffb63b, fa44cfa2801a4e65a866aa8fab126296; no voters:
I20260925 16:28:01.978596 11119 raft_consensus.cc:2781] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:28:01.978734 11119 raft_consensus.cc:482] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:28:01.978786 11119 raft_consensus.cc:3037] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:01.980630 11119 raft_consensus.cc:504] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.981596 11119 leader_election.cc:288] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [CANDIDATE]: Term 1 election: Requested vote from peers 76571d16dc574af5978a907d75ffb63b (127.0.98.2:43495), f6a63ca96ba6477b866e1cf71ee7ee9c (127.0.98.3:37403)
I20260925 16:28:01.982144 10914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32714952d99f4e68ae205e78f42b2f7c" candidate_uuid: "fa44cfa2801a4e65a866aa8fab126296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76571d16dc574af5978a907d75ffb63b"
I20260925 16:28:01.982398 11048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32714952d99f4e68ae205e78f42b2f7c" candidate_uuid: "fa44cfa2801a4e65a866aa8fab126296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c"
I20260925 16:28:01.982391 10914 raft_consensus.cc:3037] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:01.982561 11048 raft_consensus.cc:3037] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:01.984331 11048 raft_consensus.cc:2454] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa44cfa2801a4e65a866aa8fab126296 in term 1.
I20260925 16:28:01.984336 10914 raft_consensus.cc:2454] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa44cfa2801a4e65a866aa8fab126296 in term 1.
I20260925 16:28:01.984938 10713 leader_election.cc:302] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 76571d16dc574af5978a907d75ffb63b, f6a63ca96ba6477b866e1cf71ee7ee9c, fa44cfa2801a4e65a866aa8fab126296; no voters:
I20260925 16:28:01.985224 11119 raft_consensus.cc:2781] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:28:01.985522 11119 raft_consensus.cc:686] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [term 1 LEADER]: Becoming Leader. State: Replica: fa44cfa2801a4e65a866aa8fab126296, State: Running, Role: LEADER
I20260925 16:28:01.986025 11119 consensus_queue.cc:237] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } }
I20260925 16:28:01.991088 10633 catalog_manager.cc:5495] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa44cfa2801a4e65a866aa8fab126296 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fa44cfa2801a4e65a866aa8fab126296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa44cfa2801a4e65a866aa8fab126296" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } health_report { overall_health: UNKNOWN } } }
I20260925 16:28:02.097251 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:02.091688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.98.1:35965)
0925 16:28:02.091887 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:02.091892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:02.091913 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:28:02.092037 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:02.092039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:02.092147 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:02.092274 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:28:02.092605 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:02.093211 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:02.094456 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:02.094830 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:02.095028 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:02.095544 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:28:02.096420 (+ 876us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:28:02.096589 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:02.096850 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 16:28:02.096992 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:28:02.107069 11048 raft_consensus.cc:1261] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 1 FOLLOWER]: Refusing update from remote peer fa44cfa2801a4e65a866aa8fab126296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:28:02.107378 10914 raft_consensus.cc:1261] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 1 FOLLOWER]: Refusing update from remote peer fa44cfa2801a4e65a866aa8fab126296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:28:02.108111 11122 consensus_queue.cc:1035] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a63ca96ba6477b866e1cf71ee7ee9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:28:02.108469 11119 consensus_queue.cc:1035] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76571d16dc574af5978a907d75ffb63b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:28:02.133730 10633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:28:02.134028 10633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:28:02.137106 10633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d40c45051294441d979dbc78b97a067e: Sending DeleteTablet for 3 replicas of tablet 32714952d99f4e68ae205e78f42b2f7c
I20260925 16:28:02.138415 11028 tablet_service.cc:1514] Processing DeleteTablet for tablet 32714952d99f4e68ae205e78f42b2f7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:28:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56458
I20260925 16:28:02.138461 10894 tablet_service.cc:1514] Processing DeleteTablet for tablet 32714952d99f4e68ae205e78f42b2f7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:28:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50186
I20260925 16:28:02.139240 11135 tablet_replica.cc:330] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: stopping tablet replica
I20260925 16:28:02.139242 10760 tablet_service.cc:1514] Processing DeleteTablet for tablet 32714952d99f4e68ae205e78f42b2f7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:28:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39106
I20260925 16:28:02.139922 11135 raft_consensus.cc:2229] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:28:02.139997 11137 tablet_replica.cc:330] T 32714952d99f4e68ae205e78f42b2f7c P fa44cfa2801a4e65a866aa8fab126296: stopping tablet replica
I20260925 16:28:02.140079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10697
I20260925 16:28:02.141295 11135 raft_consensus.cc:2258] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:28:02.142474 11136 tablet_replica.cc:330] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: stopping tablet replica
I20260925 16:28:02.143033 11136 raft_consensus.cc:2229] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:28:02.143563 11136 raft_consensus.cc:2258] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:28:02.145169 11136 ts_tablet_manager.cc:1905] T 32714952d99f4e68ae205e78f42b2f7c P 76571d16dc574af5978a907d75ffb63b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:28:02.145162 11135 ts_tablet_manager.cc:1905] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:28:02.147526 10622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35965 (error 108)
I20260925 16:28:02.147555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10831
I20260925 16:28:02.150826 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:02.150299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35965 (local address 127.0.0.1:39126)
0925 16:28:02.150587 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:02.150689 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
W20260925 16:28:02.151209 10622 catalog_manager.cc:4584] TS fa44cfa2801a4e65a866aa8fab126296 (127.0.98.1:35965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32714952d99f4e68ae205e78f42b2f7c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35965: connect: Connection refused (error 111)
I20260925 16:28:02.151860 11135 ts_tablet_manager.cc:1918] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:28:02.151990 11135 log.cc:1196] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/32714952d99f4e68ae205e78f42b2f7c
I20260925 16:28:02.152313 11135 ts_tablet_manager.cc:1939] T 32714952d99f4e68ae205e78f42b2f7c P f6a63ca96ba6477b866e1cf71ee7ee9c: Deleting consensus metadata
I20260925 16:28:02.154096 10619 catalog_manager.cc:4841] TS f6a63ca96ba6477b866e1cf71ee7ee9c (127.0.98.3:37403): tablet 32714952d99f4e68ae205e78f42b2f7c (table test-table [id=d22e85c695ec4a40acb1f30d5aff953e]) successfully deleted
I20260925 16:28:02.155473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10965
I20260925 16:28:02.158211 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:02.157713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43495 (local address 127.0.0.1:50194)
0925 16:28:02.157971 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:02.158065 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 16:28:02.162550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 10603
2026-09-25T16:28:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2918 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:28:02 dist-test-slave-31kt krb5kdc[11142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:28:02 dist-test-slave-31kt krb5kdc[11142](info): set up 2 sockets
Sep 25 16:28:02 dist-test-slave-31kt krb5kdc[11142](info): commencing operation
krb5kdc: starting...
W20260925 16:28:03.151067 392 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:28:03 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:28:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:28:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 16:28:03.346210 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task10sHI3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/postgres -l logfile start
2026-09-25 16:28:06.396 UTC [11172] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 16:28:06.396 UTC [11172] LOG: listening on IPv4 address "127.0.98.20", port 54595
2026-09-25 16:28:06.402 UTC [11172] LOG: listening on Unix socket "/tmp/.s.PGSQL.54595"
2026-09-25 16:28:06.428 UTC [11175] LOG: database system was shut down at 2026-09-25 16:28:04 UTC
2026-09-25 16:28:06.434 UTC [11172] LOG: database system is ready to accept connections
I20260925 16:28:07.302765 392 mini_postgres.cc:96] Postgres bound to 54595
2026-09-25 16:28:07.311 UTC [11183] FATAL: database "slave" does not exist
127.0.98.20:54595 - accepting connections
I20260925 16:28:07.312028 392 mini_ranger.cc:161] Starting Ranger...
I20260925 16:28:07.329622 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 16:28:07.954598 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 16:28:07.954782 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-admin
2026-09-25 16:28:08,265 [I] DB FLAVOR :POSTGRES
2026-09-25 16:28:08,266 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:28:08,266 [I] Checking connection..
2026-09-25 16:28:08,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 16:28:08,575 [I] Checking connection passed.
2026-09-25 16:28:08,576 [I] --------- Verifying version history table ---------
2026-09-25 16:28:08,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:28:08,884 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:28:08,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:28:09,184 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:28:09,185 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 16:28:09,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 16:28:09.500 UTC [11247] WARNING: there is no transaction in progress
2026-09-25 16:28:09,508 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 16:28:09,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:28:09,813 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 16:28:09,813 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:28:09,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 16:28:10,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 16:28:10,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-31kt.c.gcp-upstream.internal','N') ;"
2026-09-25 16:28:10,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:28:11,027 [I] Table x_portal_user does not exist in database ranger
2026-09-25 16:28:11,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:28:11,332 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 16:28:11,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:28:11,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:28:11,943 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 16:28:11,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 16:28:13.192 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.212 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.233 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.253 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.273 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.295 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.363 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.373 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.394 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.407 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.421 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.441 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.456 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.466 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.476 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:13.488 UTC [11355] WARNING: there is no transaction in progress
2026-09-25 16:28:14,456 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 16:28:14,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 16:28:14,741 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 16:28:14,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:28:15,061 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 16:28:15,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:28:15,358 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 16:28:15,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:28:15,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:28:15,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-31kt.c.gcp-upstream.internal';"
2026-09-25 16:28:16,237 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 16:28:16,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 16:28:16,530 [I] DB_PATCHES have already been applied
I20260925 16:28:16.537369 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task10sHI3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 16:28:16.537460 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 16:28:16.542047 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34603
Sep 25, 2026 4:28:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:28:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 16:28:18.702791 392 mini_ranger.cc:160] Time spent starting Ranger: real 11.391s user 0.004s sys 0.012s
Sep 25, 2026 4:28:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:28:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:28:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:28:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:28:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 16:28:46.101459 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 16:28:46 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353726, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 16:28:46.202400 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 16:28:46.217193 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 16:28:47.225521 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 16:28:47.225653 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms
I20260925 16:28:47.225832 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54595
2026-09-25 16:28:47,472 [I] DB FLAVOR :POSTGRES
2026-09-25 16:28:47,472 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:28:47,472 [I] Checking connection
2026-09-25 16:28:47,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 16:28:47,779 [I] connection success
2026-09-25 16:28:47,779 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 16:28:47,780 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 16:28:47,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 16:28:48,061 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 16:28:48,061 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:28:48,062 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 16:28:48,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 16:28:48,405 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 16:28:49.014225 392 mini_ranger_kms.cc:310] Created kms service
I20260925 16:28:49.271209 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 16:28:49.329079 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 16:28:49 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353726, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:28:49.951184 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 16:28:49.951364 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 16:28:49.951390 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 16:28:49.954869 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11537
I20260925 16:28:49.955036 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54751
Sep 25, 2026 4:28:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:28:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:28:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:28:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:28:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:28:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 16:28:52.088171 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.886s user 0.000s sys 0.227s
I20260925 16:28:52.088325 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 16:28:52.088466 392 mini_ranger_kms.cc:401] 127.0.98.20:54751/kms/v1/keys
Sep 25 16:28:52 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790353732, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 16:28:53 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353732, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:28:53 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353732, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:28:56 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:28:56.628016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36219
--ranger_config_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54751/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:28:56.783982 11587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:56.784466 11587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:56.784515 11587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:56.789691 11587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:28:56.789873 11587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:56.789898 11587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:56.789918 11587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:28:56.789938 11587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:28:56.798077 11587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54751/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36219
--ranger_config_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:56.799793 11587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:56.801283 11587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:56.814227 11593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:56.814666 11592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:56.815011 11587 server_base.cc:1034] running on GCE node
W20260925 16:28:56.815147 11595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:56.816382 11587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:56.818040 11587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:56.819464 11587 hybrid_clock.cc:648] HybridClock initialized: now 1790353736819418 us; error 77 us; skew 500 ppm
Sep 25 16:28:56 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:56.828049 11587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:28:56.829697 11587 webserver.cc:460] Webserver started at http://127.0.98.62:35985/ using document root <none> and password file <none>
I20260925 16:28:56.830588 11587 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:56.830673 11587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:56.830921 11587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:28:56 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:28:56.893869 11587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "e68db22c7a2f4a3fa4b5039a8e883de6"
format_stamp: "Formatted at 2026-09-25 16:28:56 on dist-test-slave-31kt"
server_key: "b881b9db1947714882b50d62fbba3f94"
server_key_iv: "03c7c2ad0f1105ccb096c77d260cc593"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:56.894847 11587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "e68db22c7a2f4a3fa4b5039a8e883de6"
format_stamp: "Formatted at 2026-09-25 16:28:56 on dist-test-slave-31kt"
server_key: "b881b9db1947714882b50d62fbba3f94"
server_key_iv: "03c7c2ad0f1105ccb096c77d260cc593"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:56.900687 11587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:28:56.921687 11602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:56.923556 11587 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:28:56.923738 11587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "e68db22c7a2f4a3fa4b5039a8e883de6"
format_stamp: "Formatted at 2026-09-25 16:28:56 on dist-test-slave-31kt"
server_key: "b881b9db1947714882b50d62fbba3f94"
server_key_iv: "03c7c2ad0f1105ccb096c77d260cc593"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:56.923902 11587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:56.944764 11587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:56.948654 11587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:56.948900 11587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:56.959275 11587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36219
I20260925 16:28:56.959357 11654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36219 every 8 connection(s)
I20260925 16:28:56.960709 11587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:28:56.964656 11655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:56.970460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 11587
I20260925 16:28:56.970628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:28:56.970928 392 external_mini_cluster.cc:1428] Setting key 92ab93f1336d5b62a89f2748d19015be
I20260925 16:28:56.972648 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6: Bootstrap starting.
I20260925 16:28:56.976227 11655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:56.977341 11655 log.cc:824] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:56.980139 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6: No bootstrap required, opened a new log
Sep 25 16:28:56 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:28:56.985584 11655 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36219 } }
I20260925 16:28:56.985992 11655 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:56.986042 11655 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68db22c7a2f4a3fa4b5039a8e883de6, State: Initialized, Role: FOLLOWER
I20260925 16:28:56.986770 11655 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36219 } }
I20260925 16:28:56.986986 11655 raft_consensus.cc:388] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:28:56.987062 11655 raft_consensus.cc:482] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:28:56.987192 11655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:56.988797 11655 raft_consensus.cc:504] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36219 } }
I20260925 16:28:56.989303 11655 leader_election.cc:302] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68db22c7a2f4a3fa4b5039a8e883de6; no voters:
I20260925 16:28:56.989787 11655 leader_election.cc:288] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:28:56.989915 11660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:28:56.991448 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:56.972923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.0.98.62:36219)
0925 16:28:56.973510 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:56.973532 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:56.973584 (+ 52us) server_negotiation.cc:407] Connection header received
0925 16:28:56.974520 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:56.974545 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:56.975029 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:56.975463 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:56.976677 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:56.978046 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:56.979142 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:56.980024 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:56.980229 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:56.983581 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:56.983590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:56.983613 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:56.983661 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:56.986313 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:56.987159 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:56.987167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:56.987176 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:56.987344 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:56.987801 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:56.987804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:56.987806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:56.988567 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:56.988816 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:56.989662 (+ 846us) server_negotiation.cc:299] Negotiation successful
0925 16:28:56.990153 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":168,"threads_started":1}
I20260925 16:28:56.991916 11655 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:28:56.991891 11660 raft_consensus.cc:686] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [term 1 LEADER]: Becoming Leader. State: Replica: e68db22c7a2f4a3fa4b5039a8e883de6, State: Running, Role: LEADER
I20260925 16:28:56.992458 11660 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36219 } }
I20260925 16:28:56.995141 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36219 } } }
I20260925 16:28:56.995200 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68db22c7a2f4a3fa4b5039a8e883de6. Latest consensus state: current_term: 1 leader_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68db22c7a2f4a3fa4b5039a8e883de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36219 } } }
I20260925 16:28:56.995461 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:28:56.995170 11655 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:28:56.995385 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:28:56.996007 11674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:28:57.003576 11674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:28:57.011812 11674 catalog_manager.cc:1347] Generated new cluster ID: 37da3530607e45618ba36ff42cd11f05
I20260925 16:28:57.012007 11674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:28:57.021615 11674 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:28:57.023375 11674 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:28:57.030392 11674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6: Generated new TSK 0
I20260925 16:28:57.031594 11674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:28:57.875967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36219
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:28:58.042243 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:58.042709 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:58.042789 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:58.050179 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:58.050290 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:58.050391 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:28:58.059317 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:58.061321 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:58.063083 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:58.079031 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:58.079031 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:58.079070 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:58.079643 11700 server_base.cc:1034] running on GCE node
I20260925 16:28:58.080353 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:58.081271 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:58.082710 11700 hybrid_clock.cc:648] HybridClock initialized: now 1790353738082649 us; error 83 us; skew 500 ppm
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:58.087108 11700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:28:58.088886 11700 webserver.cc:460] Webserver started at http://127.0.98.1:42641/ using document root <none> and password file <none>
I20260925 16:28:58.089902 11700 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:58.090032 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:58.090345 11700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:28:58.114934 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "206acaaf181b4c72808a53036f698a24"
format_stamp: "Formatted at 2026-09-25 16:28:58 on dist-test-slave-31kt"
server_key: "b7a34e2b7753a0b42751ef14ca862742"
server_key_iv: "c5396df0872672e5bf6ad34db8fbdb36"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:58.116338 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "206acaaf181b4c72808a53036f698a24"
format_stamp: "Formatted at 2026-09-25 16:28:58 on dist-test-slave-31kt"
server_key: "b7a34e2b7753a0b42751ef14ca862742"
server_key_iv: "c5396df0872672e5bf6ad34db8fbdb36"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:58.126463 11700 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.000s
I20260925 16:28:58.152626 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:58.155535 11700 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260925 16:28:58.155920 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "206acaaf181b4c72808a53036f698a24"
format_stamp: "Formatted at 2026-09-25 16:28:58 on dist-test-slave-31kt"
server_key: "b7a34e2b7753a0b42751ef14ca862742"
server_key_iv: "c5396df0872672e5bf6ad34db8fbdb36"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:58.156297 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:58.185928 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:58.194875 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:58.195185 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:58.196283 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:28:58.206259 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:28:58.206687 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 16:28:58.206907 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:28:58.207015 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:58.221377 11700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38855
I20260925 16:28:58.222939 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:28:58.224022 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38855 every 8 connection(s)
I20260925 16:28:58.230005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 11700
I20260925 16:28:58.230176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:28:58.230536 392 external_mini_cluster.cc:1428] Setting key 9d8964015d798a9e0d7bc53ee0ac0d68
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:28:58.252259 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:58.229359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45867 (local address 127.0.98.62:36219)
0925 16:28:58.232914 (+ 3555us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:58.232919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:58.232941 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:28:58.233198 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:58.233201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:58.233263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:58.233387 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:58.235322 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.236190 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.238769 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.239238 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.239372 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:58.244135 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:58.244142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:58.244145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:58.244194 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:58.247078 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:58.248333 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:58.248340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:58.248342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:58.248507 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:58.250226 (+ 1719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:58.250233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:58.250234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:58.250611 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:58.250724 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:58.251836 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0925 16:28:58.252027 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3460,"thread_start_us":143,"threads_started":1}
I20260925 16:28:58.255287 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:58.229951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36219 (local address 127.0.98.1:45867)
0925 16:28:58.231608 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:58.231682 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:58.232824 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:58.233806 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:58.233818 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:58.234252 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:28:58.235061 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:58.235087 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.236503 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.236509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:58.238453 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:58.238488 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.239746 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.239752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:58.240096 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:58.241232 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:28:58.241263 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:28:58.243712 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:28:58.247361 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:58.247374 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:58.247408 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:58.248099 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:58.248652 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:58.248657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:58.248659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:58.249964 (+ 1305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:58.250823 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:28:58.250832 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:28:58.251294 (+ 462us) client_negotiation.cc:769] Sending connection context
0925 16:28:58.252775 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0925 16:28:58.253475 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"thread_start_us":209,"threads_started":1}
I20260925 16:28:58.257406 11833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36219
I20260925 16:28:58.257742 11833 heartbeater.cc:464] Registering TS with master...
I20260925 16:28:58.258589 11833 heartbeater.cc:511] Master 127.0.98.62:36219 requested a full tablet report, sending...
I20260925 16:28:58.261056 11619 ts_manager.cc:194] Registered new tserver with Master: 206acaaf181b4c72808a53036f698a24 (127.0.98.1:38855)
I20260925 16:28:58.263770 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:28:58.282965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36219
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 16:28:58.454432 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:58.454825 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:58.454895 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:58.460263 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:58.460457 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:58.460606 11839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:28:58.470552 11839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:58.472396 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:58.475028 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:58.492436 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:58.492751 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:58.492811 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:58.493435 11839 server_base.cc:1034] running on GCE node
I20260925 16:28:58.494241 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:58.495275 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:58.498631 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790353738498575 us; error 81 us; skew 500 ppm
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:58.503687 11839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:28:58.505214 11839 webserver.cc:460] Webserver started at http://127.0.98.2:43943/ using document root <none> and password file <none>
I20260925 16:28:58.506268 11839 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:58.506379 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:58.506616 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:28:58.523767 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "1dfafbf0ac414036893fefa7189f3a5b"
format_stamp: "Formatted at 2026-09-25 16:28:58 on dist-test-slave-31kt"
server_key: "0ebc775b15117f89e12a0823a576b014"
server_key_iv: "6af9d3fff67a5de0312ef6fa9f9cf377"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:58.524495 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1dfafbf0ac414036893fefa7189f3a5b"
format_stamp: "Formatted at 2026-09-25 16:28:58 on dist-test-slave-31kt"
server_key: "0ebc775b15117f89e12a0823a576b014"
server_key_iv: "6af9d3fff67a5de0312ef6fa9f9cf377"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:58.530402 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260925 16:28:58.558621 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:58.560274 11839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:28:58.560432 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "1dfafbf0ac414036893fefa7189f3a5b"
format_stamp: "Formatted at 2026-09-25 16:28:58 on dist-test-slave-31kt"
server_key: "0ebc775b15117f89e12a0823a576b014"
server_key_iv: "6af9d3fff67a5de0312ef6fa9f9cf377"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:58.560583 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:58.597941 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:58.602469 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:58.602754 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:58.603582 11839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:28:58.605490 11839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:28:58.605779 11839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 16:28:58.605932 11839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:28:58.606005 11839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:58.619277 11839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37087
I20260925 16:28:58.619351 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37087 every 8 connection(s)
I20260925 16:28:58.621068 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:28:58.626273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 11839
I20260925 16:28:58.626444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:28:58.626837 392 external_mini_cluster.cc:1428] Setting key 24965d713f3b55a3cb0022098f5c9a3e
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:28:58.646310 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:58.626052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40637 (local address 127.0.98.62:36219)
0925 16:28:58.626209 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:58.626215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:58.629181 (+ 2966us) server_negotiation.cc:407] Connection header received
0925 16:28:58.630641 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:58.630644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:58.630706 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:58.630759 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:58.632778 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.633639 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.635645 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.636014 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.636127 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:58.639738 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:58.639744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:58.639746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:58.639783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:58.641662 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:58.642555 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:58.642559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:58.642561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:58.642666 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:58.643136 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:58.643140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:58.643142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:58.643390 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:58.643494 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:58.645967 (+ 2473us) server_negotiation.cc:299] Negotiation successful
0925 16:28:58.646142 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:28:58.647044 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:58.625362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36219 (local address 127.0.98.2:40637)
0925 16:28:58.627797 (+ 2435us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:58.627877 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:58.630344 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:58.631251 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:58.631262 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:58.631697 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:28:58.632553 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:58.632578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.633872 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.633876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:58.635440 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:58.635454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:58.636387 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:58.636391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:58.636638 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:58.637520 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:28:58.637545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:28:58.639449 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:28:58.641912 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:58.641920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:58.641944 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:58.642390 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:58.642776 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:58.642779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:58.642781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:58.643011 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:58.643553 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:28:58.643561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:28:58.643956 (+ 395us) client_negotiation.cc:769] Sending connection context
0925 16:28:58.644190 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 16:28:58.644620 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"thread_start_us":716,"threads_started":1}
I20260925 16:28:58.649832 11969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36219
I20260925 16:28:58.650230 11969 heartbeater.cc:464] Registering TS with master...
I20260925 16:28:58.651000 11969 heartbeater.cc:511] Master 127.0.98.62:36219 requested a full tablet report, sending...
I20260925 16:28:58.654291 11619 ts_manager.cc:194] Registered new tserver with Master: 1dfafbf0ac414036893fefa7189f3a5b (127.0.98.2:37087)
I20260925 16:28:58.655189 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40637
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:28:58.706867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36219
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 16:28:58.940500 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:28:58.941110 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:28:58.941227 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:28:58.949388 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:28:58.949599 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:28:58.949739 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:28:58.959699 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:28:58.962066 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:28:58.965121 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:28:58.988307 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:58.988286 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:28:58.988581 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:28:58.992856 11974 server_base.cc:1034] running on GCE node
I20260925 16:28:58.993952 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:28:58.995034 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:28:58.998324 11974 hybrid_clock.cc:648] HybridClock initialized: now 1790353738998059 us; error 271 us; skew 500 ppm
Sep 25 16:28:58 dist-test-slave-31kt krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:28:59.004055 11974 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:28:59.005553 11974 webserver.cc:460] Webserver started at http://127.0.98.3:35193/ using document root <none> and password file <none>
I20260925 16:28:59.007038 11974 fs_manager.cc:359] Metadata directory not provided
I20260925 16:28:59.007421 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:28:59.007826 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:28:59 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:28:59.047264 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2df32026c23430d93528ea8c32f25d2"
format_stamp: "Formatted at 2026-09-25 16:28:59 on dist-test-slave-31kt"
server_key: "e8ffab915c619cc1d622729fdd3c6639"
server_key_iv: "372420a67e16429068286d440ebcbe43"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:59.049703 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2df32026c23430d93528ea8c32f25d2"
format_stamp: "Formatted at 2026-09-25 16:28:59 on dist-test-slave-31kt"
server_key: "e8ffab915c619cc1d622729fdd3c6639"
server_key_iv: "372420a67e16429068286d440ebcbe43"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:59.063306 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.000s
I20260925 16:28:59.102571 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:59.104758 11974 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260925 16:28:59.104970 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "c2df32026c23430d93528ea8c32f25d2"
format_stamp: "Formatted at 2026-09-25 16:28:59 on dist-test-slave-31kt"
server_key: "e8ffab915c619cc1d622729fdd3c6639"
server_key_iv: "372420a67e16429068286d440ebcbe43"
server_key_version: "kuduclusterkey@0"
I20260925 16:28:59.105155 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:28:59.142315 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:28:59.146852 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:28:59.150998 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:28:59.152248 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:28:59.153812 11974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:28:59.161872 11974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260925 16:28:59.162115 11974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:28:59.162178 11974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:28:59.177390 11974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45819
I20260925 16:28:59.178905 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:28:59.180284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 11974
I20260925 16:28:59.180444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:28:59.180780 392 external_mini_cluster.cc:1428] Setting key c2d581bb764bb6ebfc0858b5f7164c13
I20260925 16:28:59.191493 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45819 every 8 connection(s)
Sep 25 16:28:59 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:28:59.231514 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.202414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45543 (local address 127.0.98.62:36219)
0925 16:28:59.209269 (+ 6855us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.209274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.211662 (+ 2388us) server_negotiation.cc:407] Connection header received
0925 16:28:59.211726 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.211730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.211806 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.211897 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:59.217808 (+ 5911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.218749 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.221406 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.221846 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.222091 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.226452 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:59.226458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:59.226461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:59.226498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:59.228700 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:59.229445 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:59.229449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:59.229451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:59.229552 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:59.230009 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:59.230012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:59.230014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:59.230334 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:59.230421 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.231140 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.231317 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6737,"thread_start_us":144,"threads_started":1}
I20260925 16:28:59.233688 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.200569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36219 (local address 127.0.98.3:45543)
0925 16:28:59.209653 (+ 9084us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.209809 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:59.211208 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:59.214253 (+ 3045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:59.214271 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:59.216428 (+ 2157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:28:59.217374 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.217411 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.219039 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.219045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.221146 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.221173 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.222539 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.222547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.223021 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.223944 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:28:59.223978 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:28:59.226113 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:28:59.228891 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:59.228898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:59.228916 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:59.229309 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:59.229670 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:28:59.229673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:28:59.229676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:28:59.229897 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:28:59.230494 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:28:59.230501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:28:59.230843 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 16:28:59.232095 (+ 1252us) client_negotiation.cc:240] Negotiation successful
0925 16:28:59.232509 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7690,"thread_start_us":249,"threads_started":1}
I20260925 16:28:59.235868 12103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36219
I20260925 16:28:59.236162 12103 heartbeater.cc:464] Registering TS with master...
I20260925 16:28:59.236835 12103 heartbeater.cc:511] Master 127.0.98.62:36219 requested a full tablet report, sending...
I20260925 16:28:59.238489 11619 ts_manager.cc:194] Registered new tserver with Master: c2df32026c23430d93528ea8c32f25d2 (127.0.98.3:45819)
I20260925 16:28:59.239302 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45543
I20260925 16:28:59.246292 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:28:59.265717 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.249980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.0.98.62:36219)
0925 16:28:59.250212 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.250219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.254361 (+ 4142us) server_negotiation.cc:407] Connection header received
0925 16:28:59.254626 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.254630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.254726 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.254800 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:28:59.256026 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.256898 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.258281 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.258645 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.258731 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.261012 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:28:59.261016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:28:59.261019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:28:59.261062 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:28:59.263003 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:59.263709 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:59.263714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:59.263716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:59.263822 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:28:59.264167 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:28:59.264171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:28:59.264173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:28:59.264658 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:28:59.264792 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.265306 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.265520 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 16:28:59.268553 11833 heartbeater.cc:503] Master 127.0.98.62:36219 was elected leader, sending a full tablet report...
I20260925 16:28:59.272392 11619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:28:59.277878 11619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:28:59.305068 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.294031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38855 (local address 127.0.0.1:37350)
0925 16:28:59.296468 (+ 2437us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.296543 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:59.296732 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:59.299105 (+ 2373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:59.299115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:59.299166 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:59.299420 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.299429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.301032 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.301037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.302793 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.302809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.304251 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.304256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.304420 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.304511 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 16:28:59.304640 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 16:28:59.304841 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"thread_start_us":136,"threads_started":1}
I20260925 16:28:59.306154 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.294288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.0.98.1:38855)
0925 16:28:59.295708 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.295719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.296695 (+ 976us) server_negotiation.cc:407] Connection header received
0925 16:28:59.297580 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.297587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.297935 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.299137 (+ 1202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:59.299605 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.300855 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.303008 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.304056 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.305409 (+ 1353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.305667 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.305794 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.305962 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":163,"threads_started":1}
I20260925 16:28:59.307538 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.296797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37087 (local address 127.0.0.1:47098)
0925 16:28:59.298295 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.298317 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:59.299912 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:59.300725 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:59.300728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:59.300746 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:59.300862 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.300867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.302802 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.302806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.304250 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.304261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.306513 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.306517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.306660 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.307164 (+ 504us) client_negotiation.cc:769] Sending connection context
0925 16:28:59.307245 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:28:59.307362 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":457,"thread_start_us":130,"threads_started":1}
I20260925 16:28:59.310245 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.297092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.0.98.2:37087)
0925 16:28:59.297472 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.297481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.299457 (+ 1976us) server_negotiation.cc:407] Connection header received
0925 16:28:59.300104 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.300112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.300410 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.300551 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:59.301001 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.302652 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.304621 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.305903 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.306163 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.306344 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.307332 (+ 988us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.310023 (+ 2691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":166,"threads_started":1}
I20260925 16:28:59.311193 11767 tablet_service.cc:1467] Processing CreateTablet for tablet d567fbbe28ed462a9389ce95158c7dd1 (DEFAULT_TABLE table=test-table [id=041ce0cff97848f0986a0ec3051c383d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:28:59.312642 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.297479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.98.3:45819)
0925 16:28:59.297916 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.297928 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.301478 (+ 3550us) server_negotiation.cc:407] Connection header received
0925 16:28:59.301605 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.301612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.302940 (+ 1328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.303119 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:59.303564 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.304766 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.308871 (+ 4105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.310981 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.311164 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.311376 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.312248 (+ 872us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.312450 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260925 16:28:59.312835 11767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d567fbbe28ed462a9389ce95158c7dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:59.313262 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.297095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45819 (local address 127.0.0.1:56370)
0925 16:28:59.301166 (+ 4071us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.301194 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:59.301335 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:59.303301 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:59.303304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:59.303323 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:59.303419 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.303425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.304937 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.304941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.308624 (+ 3683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.308641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.311477 (+ 2836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.311482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.311617 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.311650 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 16:28:59.311740 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:28:59.311868 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3931,"thread_start_us":117,"threads_started":1}
I20260925 16:28:59.314565 11903 tablet_service.cc:1467] Processing CreateTablet for tablet d567fbbe28ed462a9389ce95158c7dd1 (DEFAULT_TABLE table=test-table [id=041ce0cff97848f0986a0ec3051c383d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:28:59.317188 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d567fbbe28ed462a9389ce95158c7dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:59.318471 12030 tablet_service.cc:1467] Processing CreateTablet for tablet d567fbbe28ed462a9389ce95158c7dd1 (DEFAULT_TABLE table=test-table [id=041ce0cff97848f0986a0ec3051c383d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:28:59.320210 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d567fbbe28ed462a9389ce95158c7dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:28:59.327713 12124 tablet_bootstrap.cc:492] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: Bootstrap starting.
I20260925 16:28:59.332099 12124 tablet_bootstrap.cc:654] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:59.332670 12123 tablet_bootstrap.cc:492] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: Bootstrap starting.
I20260925 16:28:59.333173 12124 log.cc:824] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:59.334692 12125 tablet_bootstrap.cc:492] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Bootstrap starting.
I20260925 16:28:59.336361 12123 tablet_bootstrap.cc:654] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:59.337713 12123 log.cc:824] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:59.340869 12124 tablet_bootstrap.cc:492] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: No bootstrap required, opened a new log
I20260925 16:28:59.341163 12124 ts_tablet_manager.cc:1397] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260925 16:28:59.341647 12125 tablet_bootstrap.cc:654] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Neither blocks nor log segments found. Creating new log.
I20260925 16:28:59.343897 12125 log.cc:824] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:28:59.345240 12123 tablet_bootstrap.cc:492] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: No bootstrap required, opened a new log
I20260925 16:28:59.345582 12123 ts_tablet_manager.cc:1397] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 16:28:59.347924 12124 raft_consensus.cc:348] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.350021 12124 raft_consensus.cc:374] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:59.350133 12124 raft_consensus.cc:729] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfafbf0ac414036893fefa7189f3a5b, State: Initialized, Role: FOLLOWER
I20260925 16:28:59.350883 12124 consensus_queue.cc:260] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.352397 12124 ts_tablet_manager.cc:1428] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: Time spent starting tablet: real 0.011s user 0.001s sys 0.008s
I20260925 16:28:59.352862 11969 heartbeater.cc:503] Master 127.0.98.62:36219 was elected leader, sending a full tablet report...
I20260925 16:28:59.354411 12123 raft_consensus.cc:348] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.354882 12123 raft_consensus.cc:374] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:59.354956 12123 raft_consensus.cc:729] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206acaaf181b4c72808a53036f698a24, State: Initialized, Role: FOLLOWER
I20260925 16:28:59.355736 12123 consensus_queue.cc:260] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.357410 12123 ts_tablet_manager.cc:1428] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260925 16:28:59.362316 12125 tablet_bootstrap.cc:492] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: No bootstrap required, opened a new log
I20260925 16:28:59.362623 12125 ts_tablet_manager.cc:1397] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20260925 16:28:59.367831 12125 raft_consensus.cc:348] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.368240 12125 raft_consensus.cc:374] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:28:59.368292 12125 raft_consensus.cc:729] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2df32026c23430d93528ea8c32f25d2, State: Initialized, Role: FOLLOWER
I20260925 16:28:59.368906 12125 consensus_queue.cc:260] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.370304 12125 ts_tablet_manager.cc:1428] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 16:28:59.370886 12103 heartbeater.cc:503] Master 127.0.98.62:36219 was elected leader, sending a full tablet report...
W20260925 16:28:59.373572 11970 tablet.cc:2369] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:28:59.425333 12129 raft_consensus.cc:482] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:28:59.425585 12129 raft_consensus.cc:504] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.427726 12129 leader_election.cc:288] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 206acaaf181b4c72808a53036f698a24 (127.0.98.1:38855), c2df32026c23430d93528ea8c32f25d2 (127.0.98.3:45819)
Sep 25 16:28:59 dist-test-slave-31kt krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:28:59.435257 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.429595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54377 (local address 127.0.98.1:38855)
0925 16:28:59.429790 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.429796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.429999 (+ 203us) server_negotiation.cc:407] Connection header received
0925 16:28:59.430229 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.430233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.430295 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.430345 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:59.430821 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.431674 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.433430 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.434019 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.434135 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.434228 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.434962 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.435066 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:28:59.435811 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.428694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38855 (local address 127.0.98.2:54377)
0925 16:28:59.429884 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.429901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:59.430096 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:59.430494 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:59.430497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:59.430530 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:59.430692 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.430698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.431857 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.431861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.433282 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.433299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.434622 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.434626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.434742 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.434831 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:28:59.435527 (+ 696us) client_negotiation.cc:240] Negotiation successful
0925 16:28:59.435618 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":37,"thread_start_us":149,"threads_started":1}
I20260925 16:28:59.436954 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d567fbbe28ed462a9389ce95158c7dd1" candidate_uuid: "1dfafbf0ac414036893fefa7189f3a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206acaaf181b4c72808a53036f698a24" is_pre_election: true
I20260925 16:28:59.437363 11787 raft_consensus.cc:2454] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfafbf0ac414036893fefa7189f3a5b in term 0.
I20260925 16:28:59.437995 11859 leader_election.cc:302] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfafbf0ac414036893fefa7189f3a5b, 206acaaf181b4c72808a53036f698a24; no voters:
I20260925 16:28:59.438324 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.429168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45819 (local address 127.0.98.2:44503)
0925 16:28:59.431310 (+ 2142us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.431327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:28:59.431584 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:28:59.433306 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:28:59.433310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:28:59.433346 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:28:59.433540 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.433548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.434604 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.434608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.435712 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:28:59.435722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.437810 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.437814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:28:59.437924 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.437949 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:28:59.438039 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 16:28:59.438122 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2035,"mutex_wait_us":21,"thread_start_us":87,"threads_started":1}
I20260925 16:28:59.438534 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.429406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44503 (local address 127.0.98.3:45819)
0925 16:28:59.429609 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.429615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.431450 (+ 1835us) server_negotiation.cc:407] Connection header received
0925 16:28:59.431749 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.431756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.431823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.431877 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:28:59.433744 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.434441 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.435871 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.436460 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.436534 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.436594 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.438157 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.438343 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:28:59.438457 12129 raft_consensus.cc:2781] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:28:59.438650 12129 raft_consensus.cc:482] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:28:59.438701 12129 raft_consensus.cc:3037] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:59.439533 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d567fbbe28ed462a9389ce95158c7dd1" candidate_uuid: "1dfafbf0ac414036893fefa7189f3a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2df32026c23430d93528ea8c32f25d2" is_pre_election: true
I20260925 16:28:59.439994 12041 raft_consensus.cc:2454] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfafbf0ac414036893fefa7189f3a5b in term 0.
I20260925 16:28:59.440543 12129 raft_consensus.cc:504] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.441016 12129 leader_election.cc:288] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [CANDIDATE]: Term 1 election: Requested vote from peers 206acaaf181b4c72808a53036f698a24 (127.0.98.1:38855), c2df32026c23430d93528ea8c32f25d2 (127.0.98.3:45819)
I20260925 16:28:59.441663 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d567fbbe28ed462a9389ce95158c7dd1" candidate_uuid: "1dfafbf0ac414036893fefa7189f3a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2df32026c23430d93528ea8c32f25d2"
I20260925 16:28:59.441874 12041 raft_consensus.cc:3037] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:59.442086 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d567fbbe28ed462a9389ce95158c7dd1" candidate_uuid: "1dfafbf0ac414036893fefa7189f3a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206acaaf181b4c72808a53036f698a24"
I20260925 16:28:59.442248 11787 raft_consensus.cc:3037] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:28:59.444752 12041 raft_consensus.cc:2454] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfafbf0ac414036893fefa7189f3a5b in term 1.
I20260925 16:28:59.444751 11787 raft_consensus.cc:2454] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfafbf0ac414036893fefa7189f3a5b in term 1.
I20260925 16:28:59.445586 11859 leader_election.cc:302] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfafbf0ac414036893fefa7189f3a5b, c2df32026c23430d93528ea8c32f25d2; no voters:
I20260925 16:28:59.446339 12129 raft_consensus.cc:2781] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:28:59.446739 12129 raft_consensus.cc:686] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [term 1 LEADER]: Becoming Leader. State: Replica: 1dfafbf0ac414036893fefa7189f3a5b, State: Running, Role: LEADER
W20260925 16:28:59.447588 12104 tablet.cc:2369] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:28:59.447811 12129 consensus_queue.cc:237] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } }
I20260925 16:28:59.453649 11618 catalog_manager.cc:5495] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dfafbf0ac414036893fefa7189f3a5b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1dfafbf0ac414036893fefa7189f3a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dfafbf0ac414036893fefa7189f3a5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 } health_report { overall_health: UNKNOWN } } }
I20260925 16:28:59.473143 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.465736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.0.98.2:37087)
0925 16:28:59.466324 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 16:28:59.466331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:28:59.466349 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:28:59.466484 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:28:59.466488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:28:59.466582 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:28:59.466725 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:28:59.467407 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.468410 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.470034 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:28:59.470423 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:28:59.470536 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:28:59.471038 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:28:59.471979 (+ 941us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:28:59.472244 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 16:28:59.472826 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 16:28:59.472932 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
W20260925 16:28:59.478029 11834 tablet.cc:2369] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:28:59.484463 12041 raft_consensus.cc:1261] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 1 FOLLOWER]: Refusing update from remote peer 1dfafbf0ac414036893fefa7189f3a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:28:59.484463 11787 raft_consensus.cc:1261] T d567fbbe28ed462a9389ce95158c7dd1 P 206acaaf181b4c72808a53036f698a24 [term 1 FOLLOWER]: Refusing update from remote peer 1dfafbf0ac414036893fefa7189f3a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:28:59.485745 12135 consensus_queue.cc:1035] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2df32026c23430d93528ea8c32f25d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:28:59.486208 12129 consensus_queue.cc:1035] T d567fbbe28ed462a9389ce95158c7dd1 P 1dfafbf0ac414036893fefa7189f3a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "206acaaf181b4c72808a53036f698a24" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:28:59.519937 12140 mvcc.cc:205] Tried to move back new op lower bound from 7333288916913352704 to 7333288916783661056. Current Snapshot: MvccSnapshot[applied={T|T < 7333288916913352704}]
I20260925 16:28:59.521880 12141 mvcc.cc:205] Tried to move back new op lower bound from 7333288916913352704 to 7333288916783661056. Current Snapshot: MvccSnapshot[applied={T|T < 7333288916913352704}]
I20260925 16:28:59.526422 11619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:28:59.526697 11619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:28:59.530692 11619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e68db22c7a2f4a3fa4b5039a8e883de6: Sending DeleteTablet for 3 replicas of tablet d567fbbe28ed462a9389ce95158c7dd1
I20260925 16:28:59.532526 11903 tablet_service.cc:1514] Processing DeleteTablet for tablet d567fbbe28ed462a9389ce95158c7dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:28:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47098
I20260925 16:28:59.535661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 11700
I20260925 16:28:59.538930 12030 tablet_service.cc:1514] Processing DeleteTablet for tablet d567fbbe28ed462a9389ce95158c7dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:28:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56370
W20260925 16:28:59.545727 11606 connection.cc:535] client connection to 127.0.98.1:38855 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 16:28:59.546000 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:28:59.546068 12151 tablet_replica.cc:330] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: stopping tablet replica
I20260925 16:28:59.546566 12151 raft_consensus.cc:2229] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:28:59.547278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 11839
I20260925 16:28:59.548408 12151 raft_consensus.cc:2258] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:28:59.553488 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.553118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38855 (local address 127.0.0.1:37356)
0925 16:28:59.553274 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.553370 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 16:28:59.553911 11606 catalog_manager.cc:4584] TS 206acaaf181b4c72808a53036f698a24 (127.0.98.1:38855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d567fbbe28ed462a9389ce95158c7dd1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38855: connect: Connection refused (error 111)
I20260925 16:28:59.554666 12151 ts_tablet_manager.cc:1905] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:28:59.558184 12151 ts_tablet_manager.cc:1918] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:28:59.558334 12151 log.cc:1196] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/d567fbbe28ed462a9389ce95158c7dd1
I20260925 16:28:59.558712 12151 ts_tablet_manager.cc:1939] T d567fbbe28ed462a9389ce95158c7dd1 P c2df32026c23430d93528ea8c32f25d2: Deleting consensus metadata
I20260925 16:28:59.560449 11606 catalog_manager.cc:4841] TS c2df32026c23430d93528ea8c32f25d2 (127.0.98.3:45819): tablet d567fbbe28ed462a9389ce95158c7dd1 (table test-table [id=041ce0cff97848f0986a0ec3051c383d]) successfully deleted
I20260925 16:28:59.562252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 11974
I20260925 16:28:59.562510 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:28:59.561989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37087 (local address 127.0.0.1:47120)
0925 16:28:59.562288 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 16:28:59.562367 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 16:28:59.571843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 11587
I20260925 16:28:59.581324 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 16:28:59.918478 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 16:28:59.918704 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 16:29:00.266073 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 16:29:00.266 UTC [11172] LOG: received smart shutdown request
2026-09-25 16:29:00.270 UTC [11172] LOG: background worker "logical replication launcher" (PID 11181) exited with exit code 1
2026-09-25 16:29:00.271 UTC [11176] LOG: shutting down
2026-09-25 16:29:00.485 UTC [11172] LOG: database system is shut down
2026-09-25T16:29:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58427 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:29:00 dist-test-slave-31kt krb5kdc[12161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:29:00 dist-test-slave-31kt krb5kdc[12161](info): set up 2 sockets
Sep 25 16:29:00 dist-test-slave-31kt krb5kdc[12161](info): commencing operation
krb5kdc: starting...
W20260925 16:29:01.578194 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:29:01 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:29:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 16:29:01.793473 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task10sHI3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/postgres -l logfile start
2026-09-25 16:29:04.930 UTC [12191] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 16:29:04.930 UTC [12191] LOG: listening on IPv4 address "127.0.98.20", port 56555
2026-09-25 16:29:04.937 UTC [12191] LOG: listening on Unix socket "/tmp/.s.PGSQL.56555"
2026-09-25 16:29:04.965 UTC [12194] LOG: database system was shut down at 2026-09-25 16:29:02 UTC
2026-09-25 16:29:04.971 UTC [12191] LOG: database system is ready to accept connections
I20260925 16:29:05.837889 392 mini_postgres.cc:96] Postgres bound to 56555
2026-09-25 16:29:05.846 UTC [12202] FATAL: database "slave" does not exist
127.0.98.20:56555 - accepting connections
I20260925 16:29:05.846935 392 mini_ranger.cc:161] Starting Ranger...
I20260925 16:29:05.864511 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 16:29:06.507679 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 16:29:06.507803 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-admin
2026-09-25 16:29:06,811 [I] DB FLAVOR :POSTGRES
2026-09-25 16:29:06,812 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:29:06,812 [I] Checking connection..
2026-09-25 16:29:06,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 16:29:07,102 [I] Checking connection passed.
2026-09-25 16:29:07,102 [I] --------- Verifying version history table ---------
2026-09-25 16:29:07,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:29:07,377 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:29:07,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:29:07,660 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 16:29:07,660 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 16:29:07,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 16:29:07.950 UTC [12266] WARNING: there is no transaction in progress
2026-09-25 16:29:07,958 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 16:29:07,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 16:29:08,270 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 16:29:08,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:29:08,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 16:29:08,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 16:29:08,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-31kt.c.gcp-upstream.internal','N') ;"
2026-09-25 16:29:09,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:29:09,455 [I] Table x_portal_user does not exist in database ranger
2026-09-25 16:29:09,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:29:09,761 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 16:29:09,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:29:10,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:29:10,362 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 16:29:10,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 16:29:11.539 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.559 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.579 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.597 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.616 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.640 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.715 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.723 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.740 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.753 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.766 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.782 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.794 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.802 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.813 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:11.822 UTC [12374] WARNING: there is no transaction in progress
2026-09-25 16:29:12,675 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 16:29:12,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 16:29:12,959 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 16:29:12,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 16:29:13,233 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 16:29:13,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 16:29:13,515 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 16:29:13,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 16:29:13,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 16:29:14,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-31kt.c.gcp-upstream.internal';"
2026-09-25 16:29:14,387 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 16:29:14,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 16:29:14,678 [I] DB_PATCHES have already been applied
I20260925 16:29:14.684738 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task10sHI3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 16:29:14.684819 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 16:29:14.689491 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53313
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:29:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 16:29:16.828222 392 mini_ranger.cc:160] Time spent starting Ranger: real 10.981s user 0.000s sys 0.014s
Sep 25, 2026 4:29:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:29:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 4:29:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:29:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:29:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 16:29:42.901012 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 16:29:42 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353782, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 16:29:43.007534 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 16:29:43.019366 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 16:29:44.156631 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 16:29:44.156720 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms
I20260925 16:29:44.156739 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56555
2026-09-25 16:29:44,339 [I] DB FLAVOR :POSTGRES
2026-09-25 16:29:44,339 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 16:29:44,339 [I] Checking connection
2026-09-25 16:29:44,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 16:29:44,646 [I] connection success
2026-09-25 16:29:44,646 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 16:29:44,646 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 16:29:44,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 16:29:44,985 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 16:29:44,985 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 16:29:44,986 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 16:29:44,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 16:29:45,375 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 16:29:46.067231 392 mini_ranger_kms.cc:310] Created kms service
I20260925 16:29:46.292701 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 16:29:46.380028 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 16:29:46 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353782, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:46.927874 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 16:29:46.928085 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task10sHI3/build/debug/bin/postgresql.jar:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task10sHI3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 16:29:46.928107 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 16:29:46.932847 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12556
I20260925 16:29:46.932997 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33983
Sep 25, 2026 4:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 16:29:49.034370 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.027s user 0.022s sys 0.145s
I20260925 16:29:49.034561 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 16:29:49.034642 392 mini_ranger_kms.cc:401] 127.0.98.20:33983/kms/v1/keys
Sep 25 16:29:49 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790353789, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 16:29:49 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353789, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:29:50 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353789, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:29:53 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:29:53.358420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46223
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46223
--ranger_config_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:29:53.495687 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:53.496002 12606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:53.496054 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:53.500471 12606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:29:53.500571 12606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:53.500590 12606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:53.500607 12606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:29:53.500620 12606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:29:53.506938 12606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33983/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46223
--ranger_config_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:53.508189 12606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:53.509361 12606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:53.519907 12612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:53.520092 12606 server_base.cc:1034] running on GCE node
W20260925 16:29:53.519907 12611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:53.520385 12614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:53.521046 12606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:53.522414 12606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:53.525908 12606 hybrid_clock.cc:648] HybridClock initialized: now 1790353793525840 us; error 66 us; skew 500 ppm
Sep 25 16:29:53 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:53.530452 12606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:29:53.531764 12606 webserver.cc:460] Webserver started at http://127.0.98.62:42101/ using document root <none> and password file <none>
I20260925 16:29:53.532387 12606 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:53.532456 12606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:29:53.532675 12606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:29:53 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:53.597108 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "de5f0a20ad8f4e72ac3734e251276c42"
format_stamp: "Formatted at 2026-09-25 16:29:53 on dist-test-slave-31kt"
server_key: "fa5a5fc8ad30fb730a772b7d8815e945"
server_key_iv: "d24f5947fba71637492dbad417bef8dc"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:53.597922 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "de5f0a20ad8f4e72ac3734e251276c42"
format_stamp: "Formatted at 2026-09-25 16:29:53 on dist-test-slave-31kt"
server_key: "fa5a5fc8ad30fb730a772b7d8815e945"
server_key_iv: "d24f5947fba71637492dbad417bef8dc"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:53.602646 12606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:29:53.626471 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:53.628455 12606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:29:53.628666 12606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "de5f0a20ad8f4e72ac3734e251276c42"
format_stamp: "Formatted at 2026-09-25 16:29:53 on dist-test-slave-31kt"
server_key: "fa5a5fc8ad30fb730a772b7d8815e945"
server_key_iv: "d24f5947fba71637492dbad417bef8dc"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:53.628849 12606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:53.645057 12606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:53.648797 12606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:53.649344 12606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:53.658396 12606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46223
I20260925 16:29:53.658439 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46223 every 8 connection(s)
I20260925 16:29:53.659765 12606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:29:53.663137 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:29:53.668247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 12606
I20260925 16:29:53.668438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:29:53.668780 392 external_mini_cluster.cc:1428] Setting key d07075e2871ad159205d0157a23fc36f
I20260925 16:29:53.672112 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Bootstrap starting.
I20260925 16:29:53.675808 12674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Neither blocks nor log segments found. Creating new log.
I20260925 16:29:53.676851 12674 log.cc:824] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Log is configured to *not* fsync() on all Append() calls
I20260925 16:29:53.680016 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: No bootstrap required, opened a new log
Sep 25 16:29:53 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:29:53.684438 12674 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:53.684872 12674 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:29:53.684940 12674 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5f0a20ad8f4e72ac3734e251276c42, State: Initialized, Role: FOLLOWER
I20260925 16:29:53.685745 12674 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:53.686026 12674 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:29:53.686120 12674 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:29:53.686233 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:29:53.687618 12674 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:53.688081 12674 leader_election.cc:302] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5f0a20ad8f4e72ac3734e251276c42; no voters:
I20260925 16:29:53.688603 12674 leader_election.cc:288] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:29:53.689081 12679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:29:53.689308 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:53.670766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.0.98.62:46223)
0925 16:29:53.671473 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:53.671498 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:53.671555 (+ 57us) server_negotiation.cc:407] Connection header received
0925 16:29:53.672431 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:53.672457 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:53.672967 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:53.673378 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:53.674910 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:53.676161 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:53.677503 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:53.678345 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:53.678507 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:53.682174 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:53.682184 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:53.682216 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:53.682274 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:53.684404 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:53.685260 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:53.685271 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:53.685281 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:53.685475 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:53.686101 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:53.686105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:53.686107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:53.686824 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:53.687053 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:53.687501 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 16:29:53.687996 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":177,"threads_started":1}
I20260925 16:29:53.689388 12679 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 LEADER]: Becoming Leader. State: Replica: de5f0a20ad8f4e72ac3734e251276c42, State: Running, Role: LEADER
I20260925 16:29:53.690162 12679 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:53.690562 12674 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:29:53.693382 12674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:29:53.693318 12681 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5f0a20ad8f4e72ac3734e251276c42. Latest consensus state: current_term: 1 leader_uuid: "de5f0a20ad8f4e72ac3734e251276c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } } }
I20260925 16:29:53.693580 12680 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5f0a20ad8f4e72ac3734e251276c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } } }
I20260925 16:29:53.693689 12681 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: This master's current role is: LEADER
I20260925 16:29:53.693708 12680 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: This master's current role is: LEADER
I20260925 16:29:53.694550 12694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:29:53.700866 12694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:29:53.710242 12694 catalog_manager.cc:1347] Generated new cluster ID: 7abc99c526de4abc92f642c55a18e599
I20260925 16:29:53.710376 12694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:29:53.726179 12694 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:29:53.728708 12694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:29:53.735296 12694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Generated new TSK 0
I20260925 16:29:53.736670 12694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:29:54.533653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46223
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:29:54.712036 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:54.712484 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:54.712563 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:54.717617 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:54.717741 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:54.717880 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:29:54.725837 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:54.727799 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:54.729544 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:54.743922 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:54.743984 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:54.743922 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:54.745821 12719 server_base.cc:1034] running on GCE node
I20260925 16:29:54.746598 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:54.747658 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:54.751026 12719 hybrid_clock.cc:648] HybridClock initialized: now 1790353794750967 us; error 76 us; skew 500 ppm
Sep 25 16:29:54 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:54.756065 12719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:29:54.757313 12719 webserver.cc:460] Webserver started at http://127.0.98.1:35627/ using document root <none> and password file <none>
I20260925 16:29:54.758005 12719 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:54.758078 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:29:54.758291 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:29:54 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:54.780743 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "0112a89312644ba390b4ef5b7f6aa7ce"
format_stamp: "Formatted at 2026-09-25 16:29:54 on dist-test-slave-31kt"
server_key: "3c5366eb053cd2b10e6af9c3f61f84c5"
server_key_iv: "528b58f2c8d7a63b5e9cc036e97764f3"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:54.782042 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0112a89312644ba390b4ef5b7f6aa7ce"
format_stamp: "Formatted at 2026-09-25 16:29:54 on dist-test-slave-31kt"
server_key: "3c5366eb053cd2b10e6af9c3f61f84c5"
server_key_iv: "528b58f2c8d7a63b5e9cc036e97764f3"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:54.790097 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260925 16:29:54.814327 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:54.817387 12719 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260925 16:29:54.817822 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "0112a89312644ba390b4ef5b7f6aa7ce"
format_stamp: "Formatted at 2026-09-25 16:29:54 on dist-test-slave-31kt"
server_key: "3c5366eb053cd2b10e6af9c3f61f84c5"
server_key_iv: "528b58f2c8d7a63b5e9cc036e97764f3"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:54.818156 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:54.838663 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:54.844231 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:54.844663 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:54.845817 12719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:29:54.847520 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:29:54.847596 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:54.847675 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:29:54.847725 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 16:29:54.862357 12719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36369
I20260925 16:29:54.864173 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:29:54.868440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 12719
I20260925 16:29:54.868646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:29:54.869242 392 external_mini_cluster.cc:1428] Setting key 16794cc12f16f89b2440d3e9dc35aeef
I20260925 16:29:54.874562 12851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36369 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 16:29:54 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:29:54.910513 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:54.878247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35045 (local address 127.0.98.62:46223)
0925 16:29:54.890742 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:54.890748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:54.891304 (+ 556us) server_negotiation.cc:407] Connection header received
0925 16:29:54.892861 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:54.892866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:54.892942 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:54.893036 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:54.895439 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:54.896267 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:54.898237 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:54.898617 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:54.898710 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:54.904743 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:54.904748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:54.904751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:54.904788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:54.907366 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:54.908273 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:54.908278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:54.908280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:54.908395 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:54.908868 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:54.908871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:54.908873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:54.909132 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:54.909215 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:54.910128 (+ 913us) server_negotiation.cc:299] Negotiation successful
0925 16:29:54.910313 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12384,"thread_start_us":159,"threads_started":1}
I20260925 16:29:54.912727 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:54.879529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46223 (local address 127.0.98.1:35045)
0925 16:29:54.890434 (+ 10905us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:54.890799 (+ 365us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:54.892411 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:54.893460 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:54.893472 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:54.893897 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:29:54.895193 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:54.895220 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:54.896479 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:54.896483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:54.898050 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:54.898068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:54.901302 (+ 3234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:54.901308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:54.901595 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:54.902509 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:29:54.902534 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:29:54.904440 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:29:54.907624 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:54.907639 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:54.907664 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:54.908110 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:54.908513 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:54.908515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:54.908518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:54.908738 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:54.909305 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:29:54.909313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:29:54.909800 (+ 487us) client_negotiation.cc:769] Sending connection context
0925 16:29:54.911076 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0925 16:29:54.911540 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10449,"thread_start_us":208,"threads_started":1}
I20260925 16:29:54.914831 12852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46223
I20260925 16:29:54.915136 12852 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:54.916152 12852 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:54.918532 12638 ts_manager.cc:194] Registered new tserver with Master: 0112a89312644ba390b4ef5b7f6aa7ce (127.0.98.1:36369)
I20260925 16:29:54.920275 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:29:54.946857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46223
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:29:54 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790353794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 16:29:55.112826 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:55.113250 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:55.113318 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:55.118451 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:55.118623 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:55.118762 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:29:55.126641 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:55.128521 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:55.130097 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:55.143537 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:55.143767 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:55.143537 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:55.145542 12858 server_base.cc:1034] running on GCE node
I20260925 16:29:55.146392 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:55.147621 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:55.150957 12858 hybrid_clock.cc:648] HybridClock initialized: now 1790353795150900 us; error 81 us; skew 500 ppm
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:55.159706 12858 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:29:55.161365 12858 webserver.cc:460] Webserver started at http://127.0.98.2:45503/ using document root <none> and password file <none>
I20260925 16:29:55.162335 12858 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:55.162477 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:29:55.162793 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:55.186209 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "8bf8ab4ef38249adbff4d3861a539d5e"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "f0b24a5905865f2aec43c19fb10b7a78"
server_key_iv: "2792291fb744fc244352daa34104b2c0"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:55.187371 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8bf8ab4ef38249adbff4d3861a539d5e"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "f0b24a5905865f2aec43c19fb10b7a78"
server_key_iv: "2792291fb744fc244352daa34104b2c0"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:55.194411 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260925 16:29:55.224306 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:55.229811 12858 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260925 16:29:55.230221 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8bf8ab4ef38249adbff4d3861a539d5e"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "f0b24a5905865f2aec43c19fb10b7a78"
server_key_iv: "2792291fb744fc244352daa34104b2c0"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:55.230590 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:55.275640 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:55.280593 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:55.280977 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:55.282361 12858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:29:55.284204 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:29:55.284401 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 16:29:55.284586 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:29:55.284667 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:55.302652 12858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39817
I20260925 16:29:55.304705 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:29:55.306143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 12858
I20260925 16:29:55.306326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:29:55.306905 392 external_mini_cluster.cc:1428] Setting key da9860732fac7500c669ebb59b215052
I20260925 16:29:55.307519 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39817 every 8 connection(s)
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:29:55.332799 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:55.311156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46223 (local address 127.0.98.2:59847)
0925 16:29:55.312164 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:55.312292 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:55.313798 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:55.314956 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:55.314970 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:55.315493 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:29:55.316510 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:55.316540 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.317950 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.317954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:55.319663 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:55.319683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.320656 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.320660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:55.320943 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:55.321885 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:29:55.321914 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:29:55.324663 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:29:55.327581 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:55.327593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:55.327675 (+ 82us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:55.328273 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:55.328749 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:55.328752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:55.328755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:55.329032 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:55.329712 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:29:55.329721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:29:55.330287 (+ 566us) client_negotiation.cc:769] Sending connection context
0925 16:29:55.330567 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 16:29:55.331125 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":230,"threads_started":1}
I20260925 16:29:55.334416 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:55.313500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59847 (local address 127.0.98.62:46223)
0925 16:29:55.313707 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:55.313714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:55.313733 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:29:55.314180 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:55.314185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:55.314270 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:55.314325 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:55.316808 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.317690 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.319866 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.320254 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.320364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:55.325031 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:55.325036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:55.325039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:55.325084 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:55.327290 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:55.328465 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:55.328470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:55.328472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:55.328604 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:55.329157 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:55.329160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:55.329162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:55.329522 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:55.329641 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:55.333930 (+ 4289us) server_negotiation.cc:299] Negotiation successful
0925 16:29:55.334182 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 16:29:55.335985 12988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46223
I20260925 16:29:55.336400 12988 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:55.337304 12988 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:55.339676 12638 ts_manager.cc:194] Registered new tserver with Master: 8bf8ab4ef38249adbff4d3861a539d5e (127.0.98.2:39817)
I20260925 16:29:55.340641 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:29:55.372112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46223
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:29:55.571395 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:55.572346 12993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:55.572489 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:55.579694 12993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:55.579932 12993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:55.580070 12993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:29:55.590405 12993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:55.592227 12993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:55.593844 12993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:55.609115 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:55.609359 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:55.611204 12993 server_base.cc:1034] running on GCE node
W20260925 16:29:55.615032 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:55.615922 12993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:55.616886 12993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:55.632432 12993 hybrid_clock.cc:648] HybridClock initialized: now 1790353795632214 us; error 273 us; skew 500 ppm
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:55.637296 12993 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:29:55.638834 12993 webserver.cc:460] Webserver started at http://127.0.98.3:35763/ using document root <none> and password file <none>
I20260925 16:29:55.639721 12993 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:55.639889 12993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:29:55.640213 12993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:55.683207 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "e38eeb6a1ba9445e9642b49845fac106"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "0d0689a8045613b3f76dbd9782bc8cf9"
server_key_iv: "0288d28432bc57367af6162f3ee78b34"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:55.684203 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e38eeb6a1ba9445e9642b49845fac106"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "0d0689a8045613b3f76dbd9782bc8cf9"
server_key_iv: "0288d28432bc57367af6162f3ee78b34"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:55.700868 12993 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.000s
I20260925 16:29:55.740738 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:55.743587 12993 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260925 16:29:55.743907 12993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "e38eeb6a1ba9445e9642b49845fac106"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "0d0689a8045613b3f76dbd9782bc8cf9"
server_key_iv: "0288d28432bc57367af6162f3ee78b34"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:55.744205 12993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:55.790915 12993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:55.796115 12993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:55.796471 12993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:55.797333 12993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:29:55.798722 12993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:29:55.798831 12993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 16:29:55.798920 12993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:29:55.798974 12993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:55.810240 12993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36051
I20260925 16:29:55.811569 12993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:29:55.820508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 12993
I20260925 16:29:55.820652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:29:55.820963 392 external_mini_cluster.cc:1428] Setting key 272ca3822e7c3999dd4797bda896a6d3
I20260925 16:29:55.821733 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36051 every 8 connection(s)
Sep 25 16:29:55 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:29:55.846877 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:55.826813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34937 (local address 127.0.98.62:46223)
0925 16:29:55.826971 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:55.826976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:55.830529 (+ 3553us) server_negotiation.cc:407] Connection header received
0925 16:29:55.832011 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:55.832016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:55.832097 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:55.832182 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:55.833840 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.834511 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.836428 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.836697 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.836796 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:55.840965 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:55.840971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:55.840974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:55.841030 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:55.843201 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:55.844234 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:55.844240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:55.844242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:55.844367 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:55.844941 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:55.844946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:55.844948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:55.845277 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:55.845363 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:55.846459 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0925 16:29:55.846669 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:29:55.849647 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:55.827825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46223 (local address 127.0.98.3:34937)
0925 16:29:55.830254 (+ 2429us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:55.830343 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:55.831706 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:55.832451 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:55.832461 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:55.832856 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:29:55.833611 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:55.833637 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.834670 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.834675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:55.836288 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:55.836302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:55.836895 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:55.836900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:55.837140 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:55.838093 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:29:55.838127 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:29:55.840597 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:29:55.843440 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:55.843451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:55.843486 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:55.844050 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:55.844508 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:55.844512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:55.844514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:55.844799 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:55.845471 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:29:55.845482 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:29:55.846073 (+ 591us) client_negotiation.cc:769] Sending connection context
0925 16:29:55.847456 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0925 16:29:55.848054 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"thread_start_us":240,"threads_started":1}
I20260925 16:29:55.852335 13122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46223
I20260925 16:29:55.852744 13122 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:55.853657 13122 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:55.855682 12638 ts_manager.cc:194] Registered new tserver with Master: e38eeb6a1ba9445e9642b49845fac106 (127.0.98.3:36051)
I20260925 16:29:55.856674 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34937
I20260925 16:29:55.861843 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:29:55.863085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 12719
I20260925 16:29:55.872959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 12858
I20260925 16:29:55.880661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 12993
I20260925 16:29:55.887257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 12606
I20260925 16:29:55.896857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46223
--webserver_interface=127.0.98.62
--webserver_port=42101
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46223
--ranger_config_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:29:56.031810 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:56.032152 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:56.032194 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:56.036767 13125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:29:56.036868 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:56.036888 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:56.036904 13125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:29:56.036919 13125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:29:56.043372 13125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46223
--ranger_config_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:56.044535 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:56.045666 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:56.055920 13131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:56.056133 13125 server_base.cc:1034] running on GCE node
W20260925 16:29:56.055989 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:56.055894 13133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:56.056985 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:56.058379 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:56.059610 13125 hybrid_clock.cc:648] HybridClock initialized: now 1790353796059573 us; error 69 us; skew 500 ppm
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:56.063020 13125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:29:56.064172 13125 webserver.cc:460] Webserver started at http://127.0.98.62:42101/ using document root <none> and password file <none>
I20260925 16:29:56.064805 13125 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:56.064898 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:56.134474 13125 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:29:56.138199 13141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:56.140336 13125 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:29:56.140565 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "de5f0a20ad8f4e72ac3734e251276c42"
format_stamp: "Formatted at 2026-09-25 16:29:53 on dist-test-slave-31kt"
server_key: "fa5a5fc8ad30fb730a772b7d8815e945"
server_key_iv: "d24f5947fba71637492dbad417bef8dc"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:56.141536 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:56.162083 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:56.167073 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:56.167333 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:56.178586 13125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46223
I20260925 16:29:56.178653 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46223 every 8 connection(s)
I20260925 16:29:56.180375 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:29:56.185971 13194 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:29:56.187990 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Bootstrap starting.
I20260925 16:29:56.188104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13125
I20260925 16:29:56.188670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36369
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46223
--webserver_port=35627
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:29:56.208472 13194 log.cc:824] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Log is configured to *not* fsync() on all Append() calls
I20260925 16:29:56.217276 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:29:56.217738 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Bootstrap complete.
I20260925 16:29:56.222792 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:56.223390 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5f0a20ad8f4e72ac3734e251276c42, State: Initialized, Role: FOLLOWER
I20260925 16:29:56.224298 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:56.224494 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:29:56.224571 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:29:56.224676 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:29:56.227841 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:56.228286 13194 leader_election.cc:302] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5f0a20ad8f4e72ac3734e251276c42; no voters:
I20260925 16:29:56.228637 13194 leader_election.cc:288] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:29:56.229012 13199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:29:56.229288 13199 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [term 2 LEADER]: Becoming Leader. State: Replica: de5f0a20ad8f4e72ac3734e251276c42, State: Running, Role: LEADER
I20260925 16:29:56.230149 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:29:56.230453 13199 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } }
I20260925 16:29:56.232752 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de5f0a20ad8f4e72ac3734e251276c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } } }
I20260925 16:29:56.232973 13194 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 16:29:56.233536 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: This master's current role is: LEADER
I20260925 16:29:56.233799 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5f0a20ad8f4e72ac3734e251276c42. Latest consensus state: current_term: 2 leader_uuid: "de5f0a20ad8f4e72ac3734e251276c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5f0a20ad8f4e72ac3734e251276c42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46223 } } }
I20260925 16:29:56.233887 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42 [sys.catalog]: This master's current role is: LEADER
I20260925 16:29:56.235462 13212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:29:56.244568 13212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:29:56.245718 13212 catalog_manager.cc:1259] Loaded cluster ID: 7abc99c526de4abc92f642c55a18e599
I20260925 16:29:56.245848 13212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:29:56.249282 13212 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:29:56.250414 13212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Loaded TSK: 0
I20260925 16:29:56.251940 13212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:29:56.371578 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:56.372015 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:56.372093 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:56.379618 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:56.379752 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:56.379848 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:29:56.390425 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35627
--tserver_master_addrs=127.0.98.62:46223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:56.392115 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:56.393784 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:56.410456 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:56.410555 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:56.411121 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:56.412539 13196 server_base.cc:1034] running on GCE node
I20260925 16:29:56.413209 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:56.414254 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:56.415474 13196 hybrid_clock.cc:648] HybridClock initialized: now 1790353796415450 us; error 52 us; skew 500 ppm
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:56.420445 13196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:29:56.422005 13196 webserver.cc:460] Webserver started at http://127.0.98.1:35627/ using document root <none> and password file <none>
I20260925 16:29:56.422741 13196 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:56.422852 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:56.452785 13196 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 16:29:56.458570 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:56.461081 13196 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260925 16:29:56.461290 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "0112a89312644ba390b4ef5b7f6aa7ce"
format_stamp: "Formatted at 2026-09-25 16:29:54 on dist-test-slave-31kt"
server_key: "3c5366eb053cd2b10e6af9c3f61f84c5"
server_key_iv: "528b58f2c8d7a63b5e9cc036e97764f3"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:56.462200 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:56.506953 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:56.511830 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:56.512106 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:56.513104 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:29:56.514545 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:29:56.514614 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:56.514670 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:29:56.514709 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:56.527990 13196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36369
I20260925 16:29:56.528010 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36369 every 8 connection(s)
I20260925 16:29:56.529819 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:29:56.532145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13196
I20260925 16:29:56.532850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39817
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46223
--webserver_port=45503
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:29:56.564841 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:56.537917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46223 (local address 127.0.98.1:41163)
0925 16:29:56.539060 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:56.539131 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:56.540551 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:56.543904 (+ 3353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:56.543918 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:56.544393 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:29:56.545168 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:56.545193 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.547007 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.547011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:56.548270 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:56.548291 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.549551 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.549557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:56.549843 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:56.550785 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:29:56.550813 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:29:56.553143 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:29:56.558725 (+ 5582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:56.558738 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:56.558764 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:56.559398 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:56.560169 (+ 771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:56.560174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:56.560176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:56.560448 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:56.561743 (+ 1295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:29:56.561791 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:29:56.562391 (+ 600us) client_negotiation.cc:769] Sending connection context
0925 16:29:56.562743 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 16:29:56.563315 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":197,"threads_started":1}
I20260925 16:29:56.567510 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:56.537847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41163 (local address 127.0.98.62:46223)
0925 16:29:56.538697 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:56.538722 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:56.541108 (+ 2386us) server_negotiation.cc:407] Connection header received
0925 16:29:56.541635 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:56.541651 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:56.542437 (+ 786us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:56.543015 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:56.545485 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.546775 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.548516 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.549281 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.549495 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:56.553901 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:56.553913 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:56.553941 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:56.554010 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:56.558270 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:56.559687 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:56.559697 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:56.559705 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:56.559997 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:56.560672 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:56.560679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:56.560681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:56.561479 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:56.561695 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:56.565724 (+ 4029us) server_negotiation.cc:299] Negotiation successful
0925 16:29:56.566279 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":166,"threads_started":1}
I20260925 16:29:56.572275 13356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46223
I20260925 16:29:56.572650 13356 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:56.573323 13356 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
W20260925 16:29:56.576727 13356 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46223 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:29:56.576992 13356 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:56.577320 13356 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:56.578850 13356 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:56.579170 13356 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
W20260925 16:29:56.580119 13356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:29:56.734112 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:56.734683 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:56.734791 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:56.742164 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:56.742348 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:56.742479 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:29:56.754108 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45503
--tserver_master_addrs=127.0.98.62:46223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:56.755818 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:56.757473 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:56.772300 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:56.772336 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:56.772300 13366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:56.772994 13358 server_base.cc:1034] running on GCE node
I20260925 16:29:56.773576 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:56.774612 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:56.777992 13358 hybrid_clock.cc:648] HybridClock initialized: now 1790353796777879 us; error 110 us; skew 500 ppm
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:56.783381 13358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:29:56.785076 13358 webserver.cc:460] Webserver started at http://127.0.98.2:45503/ using document root <none> and password file <none>
I20260925 16:29:56.786181 13358 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:56.786397 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:56.815449 13358 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:29:56.819943 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:56.822793 13358 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260925 16:29:56.823019 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8bf8ab4ef38249adbff4d3861a539d5e"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "f0b24a5905865f2aec43c19fb10b7a78"
server_key_iv: "2792291fb744fc244352daa34104b2c0"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:56.823964 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:56.888165 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:56.892666 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:56.892925 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:56.893839 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:29:56.895682 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:29:56.895824 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 16:29:56.895933 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:29:56.895991 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:56.909579 13358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39817
I20260925 16:29:56.909657 13490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39817 every 8 connection(s)
I20260925 16:29:56.911369 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:29:56.918550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13358
I20260925 16:29:56.919166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36051
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46223
--webserver_port=35763
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:29:56 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:29:56.932862 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:56.916503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46461 (local address 127.0.98.62:46223)
0925 16:29:56.916662 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:56.916667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:56.917181 (+ 514us) server_negotiation.cc:407] Connection header received
0925 16:29:56.918962 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:56.918967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:56.919059 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:56.919123 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:56.921096 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.921995 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.924165 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.924442 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.924560 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:56.928196 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:56.928200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:56.928203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:56.928242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:56.929907 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:56.930585 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:56.930590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:56.930592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:56.930697 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:56.931160 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:56.931165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:56.931167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:56.931467 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:56.931568 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:56.932448 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 16:29:56.932647 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:29:56.938169 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:56.915054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46223 (local address 127.0.98.2:46461)
0925 16:29:56.916990 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:56.917057 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:56.918644 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:56.919615 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:56.919625 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:56.920024 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:29:56.920792 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:56.920814 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.922240 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.922245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:56.923967 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:56.923988 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:56.924788 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:56.924791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:56.925019 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:56.925950 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:29:56.925972 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:29:56.927878 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:29:56.930055 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:56.930065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:56.930093 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:56.930469 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:56.930819 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:56.930822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:56.930824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:56.931013 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:56.931580 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:29:56.931589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:29:56.932140 (+ 551us) client_negotiation.cc:769] Sending connection context
0925 16:29:56.932396 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 16:29:56.932860 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1679,"thread_start_us":216,"threads_started":1}
I20260925 16:29:56.940105 13491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46223
I20260925 16:29:56.940452 13491 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:56.941188 13491 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
W20260925 16:29:56.943328 13491 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46223 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 16:29:56.943581 13491 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:56.943892 13491 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:56.945107 13491 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:56.945399 13491 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
W20260925 16:29:56.946497 13491 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:29:57.106237 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:29:57.106669 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:29:57.106741 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:29:57.113368 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:29:57.113483 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:29:57.113574 13494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:29:57.121356 13494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35763
--tserver_master_addrs=127.0.98.62:46223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:29:57.123279 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:29:57.124878 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:29:57.137337 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:29:57.137658 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:57.137722 13494 server_base.cc:1034] running on GCE node
W20260925 16:29:57.139017 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:29:57.139833 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:29:57.140810 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:29:57.142261 13494 hybrid_clock.cc:648] HybridClock initialized: now 1790353797142236 us; error 54 us; skew 500 ppm
Sep 25 16:29:57 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:57.146381 13494 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:29:57.147764 13494 webserver.cc:460] Webserver started at http://127.0.98.3:35763/ using document root <none> and password file <none>
I20260925 16:29:57.148691 13494 fs_manager.cc:359] Metadata directory not provided
I20260925 16:29:57.148891 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 16:29:57 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:57.178663 13494 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 16:29:57.185976 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:57.193014 13494 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260925 16:29:57.193284 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "e38eeb6a1ba9445e9642b49845fac106"
format_stamp: "Formatted at 2026-09-25 16:29:55 on dist-test-slave-31kt"
server_key: "0d0689a8045613b3f76dbd9782bc8cf9"
server_key_iv: "0288d28432bc57367af6162f3ee78b34"
server_key_version: "kuduclusterkey@0"
I20260925 16:29:57.194468 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:29:57.218279 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:29:57.222806 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:29:57.223078 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:29:57.223946 13494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:29:57.225142 13494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:29:57.225198 13494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:57.225247 13494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:29:57.225262 13494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:29:57.237874 13494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36051
I20260925 16:29:57.238262 13630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36051 every 8 connection(s)
I20260925 16:29:57.239553 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:29:57.241353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13494
Sep 25 16:29:57 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:29:57.259227 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:57.242560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.0.98.62:46223)
0925 16:29:57.242758 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:57.242764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:57.243618 (+ 854us) server_negotiation.cc:407] Connection header received
0925 16:29:57.243729 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:57.243733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:57.243817 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:57.244000 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:57.245593 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.246536 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.247930 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.248327 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.248426 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:57.251000 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:57.251004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:57.251007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:57.251076 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:57.253360 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:57.256832 (+ 3472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:57.256838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:57.256841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:57.256974 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:57.257404 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:57.257407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:57.257409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:57.257701 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:57.258059 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:57.258769 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 16:29:57.258990 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:29:57.263921 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:57.245175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52745 (local address 127.0.98.62:46223)
0925 16:29:57.245521 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:57.245534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:57.245945 (+ 411us) server_negotiation.cc:407] Connection header received
0925 16:29:57.247400 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:57.247404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:57.247535 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:57.247635 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:57.249984 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.250591 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.252385 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.252703 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.252824 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:57.257676 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:57.257681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:57.257693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:57.257720 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:57.259722 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:57.260850 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:57.260856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:57.260858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:57.261053 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:57.261982 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:57.261986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:57.261988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:57.262345 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:57.262455 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:57.263419 (+ 964us) server_negotiation.cc:299] Negotiation successful
0925 16:29:57.263663 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260925 16:29:57.265528 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:57.243619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46223 (local address 127.0.98.3:52745)
0925 16:29:57.245715 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:57.245818 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:57.246962 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:57.248160 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:57.248175 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:57.248743 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:29:57.249744 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:57.249800 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.250744 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.250748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:57.252245 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:57.252264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.253130 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.254263 (+ 1133us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:57.254583 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:57.255606 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:29:57.255631 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:29:57.257404 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:29:57.260052 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:57.260063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:57.260095 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:57.260645 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:57.261430 (+ 785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:29:57.261433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:29:57.261435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:29:57.261845 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:29:57.262474 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:29:57.262485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:29:57.262931 (+ 446us) client_negotiation.cc:769] Sending connection context
0925 16:29:57.263223 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 16:29:57.263930 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"thread_start_us":264,"threads_started":1}
I20260925 16:29:57.267763 13631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46223
I20260925 16:29:57.268225 13631 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:57.269078 13631 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:57.271595 13153 ts_manager.cc:194] Registered new tserver with Master: e38eeb6a1ba9445e9642b49845fac106 (127.0.98.3:36051)
I20260925 16:29:57.273906 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52745
I20260925 16:29:57.580416 13356 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:57.580858 13356 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:57.582444 13153 ts_manager.cc:194] Registered new tserver with Master: 0112a89312644ba390b4ef5b7f6aa7ce (127.0.98.1:36369)
I20260925 16:29:57.583487 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41163
Sep 25 16:29:57 dist-test-slave-31kt krb5kdc[12161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790353797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:29:57.946785 13491 heartbeater.cc:464] Registering TS with master...
I20260925 16:29:57.947178 13491 heartbeater.cc:511] Master 127.0.98.62:46223 requested a full tablet report, sending...
I20260925 16:29:57.948192 13153 ts_manager.cc:194] Registered new tserver with Master: 8bf8ab4ef38249adbff4d3861a539d5e (127.0.98.2:39817)
I20260925 16:29:57.949079 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46461
I20260925 16:29:57.959228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:29:57.976477 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:57.962459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.0.98.62:46223)
0925 16:29:57.964921 (+ 2462us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:57.964926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:57.965375 (+ 449us) server_negotiation.cc:407] Connection header received
0925 16:29:57.965549 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:57.965553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:57.965642 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:57.965832 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:29:57.966770 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.967708 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.969067 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:57.969453 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:57.969551 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:57.971668 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:29:57.971673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:29:57.971677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:29:57.971729 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:29:57.973876 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:57.974604 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:57.974609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:57.974611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:57.974721 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:29:57.975081 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:29:57.975085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:29:57.975087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:29:57.975396 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:29:57.975530 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:57.976044 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 16:29:57.976237 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2337,"thread_start_us":103,"threads_started":1}
I20260925 16:29:57.983574 13153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:29:57.988538 13153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:29:58.014405 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.005662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36369 (local address 127.0.0.1:41880)
0925 16:29:58.007170 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.007178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:58.007256 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:58.008280 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:58.008283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:58.008299 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:29:58.008386 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.008391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.010231 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.010235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.011999 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.012011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.013603 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.013607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.013747 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.013825 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 16:29:58.013967 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 16:29:58.014163 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"mutex_wait_us":20,"thread_start_us":649,"threads_started":1}
I20260925 16:29:58.015420 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.006069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.0.98.1:36369)
0925 16:29:58.006361 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:58.006370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:58.007276 (+ 906us) server_negotiation.cc:407] Connection header received
0925 16:29:58.007379 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:58.007385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:58.007658 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:58.007820 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:29:58.008512 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.010111 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.012171 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.013182 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.013305 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.013522 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:58.015002 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0925 16:29:58.015179 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260925 16:29:58.015645 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.004312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39817 (local address 127.0.0.1:49966)
0925 16:29:58.006943 (+ 2631us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.006956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:58.007038 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:58.007967 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:58.007974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:58.007999 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:29:58.008165 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.008172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.010491 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.010494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.011999 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.012011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.015182 (+ 3171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.015185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.015311 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.015339 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 16:29:58.015392 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 16:29:58.015501 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2566,"mutex_wait_us":132,"thread_start_us":197,"threads_started":1}
I20260925 16:29:58.017570 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.004637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.0.98.2:39817)
0925 16:29:58.007536 (+ 2899us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:58.007543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:58.007566 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:29:58.007642 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:58.007646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:58.007801 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:58.008307 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:29:58.008498 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.010360 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.012147 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.013182 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.016918 (+ 3736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.017170 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:58.017240 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 16:29:58.017381 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2758,"thread_start_us":164,"threads_started":1}
I20260925 16:29:58.020001 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 877dd20a4874430ca02a6ece2e162fb6 (DEFAULT_TABLE table=test-table [id=16428ca694574948a4e29568db89bb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:29:58.020853 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 877dd20a4874430ca02a6ece2e162fb6 (DEFAULT_TABLE table=test-table [id=16428ca694574948a4e29568db89bb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:29:58.021581 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dd20a4874430ca02a6ece2e162fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:29:58.021829 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dd20a4874430ca02a6ece2e162fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:29:58.038988 13657 tablet_bootstrap.cc:492] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce: Bootstrap starting.
I20260925 16:29:58.039218 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.004247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36051 (local address 127.0.0.1:46872)
0925 16:29:58.005412 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.005729 (+ 317us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:58.005917 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:58.034645 (+ 28728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:58.034649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:58.034677 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:29:58.034843 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.034850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.036074 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.036078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.037243 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.037256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.038705 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.038709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.038817 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.038852 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:29:58.038910 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 16:29:58.039026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":825,"threads_started":1}
I20260925 16:29:58.040364 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.004628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.0.98.3:36051)
0925 16:29:58.032056 (+ 27428us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:58.032071 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:58.032124 (+ 53us) server_negotiation.cc:407] Connection header received
0925 16:29:58.033839 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:58.033848 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:58.034452 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:58.034989 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:29:58.035133 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.035947 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.037431 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.038551 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.039760 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.039975 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:58.040041 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 16:29:58.040181 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8764,"thread_start_us":75,"threads_started":1}
I20260925 16:29:58.044374 13657 tablet_bootstrap.cc:654] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce: Neither blocks nor log segments found. Creating new log.
I20260925 16:29:58.045634 13657 log.cc:824] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce: Log is configured to *not* fsync() on all Append() calls
I20260925 16:29:58.045182 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 877dd20a4874430ca02a6ece2e162fb6 (DEFAULT_TABLE table=test-table [id=16428ca694574948a4e29568db89bb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:29:58.046470 13658 tablet_bootstrap.cc:492] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Bootstrap starting.
I20260925 16:29:58.046756 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dd20a4874430ca02a6ece2e162fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:29:58.050794 13658 tablet_bootstrap.cc:654] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Neither blocks nor log segments found. Creating new log.
I20260925 16:29:58.051889 13658 log.cc:824] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Log is configured to *not* fsync() on all Append() calls
I20260925 16:29:58.054637 13657 tablet_bootstrap.cc:492] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce: No bootstrap required, opened a new log
I20260925 16:29:58.054963 13657 ts_tablet_manager.cc:1397] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260925 16:29:58.063410 13658 tablet_bootstrap.cc:492] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: No bootstrap required, opened a new log
I20260925 16:29:58.063750 13658 ts_tablet_manager.cc:1397] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260925 16:29:58.069219 13661 tablet_bootstrap.cc:492] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Bootstrap starting.
I20260925 16:29:58.060264 13657 raft_consensus.cc:348] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.069837 13657 raft_consensus.cc:374] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:29:58.069948 13657 raft_consensus.cc:729] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0112a89312644ba390b4ef5b7f6aa7ce, State: Initialized, Role: FOLLOWER
I20260925 16:29:58.069288 13658 raft_consensus.cc:348] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.070577 13658 raft_consensus.cc:374] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:29:58.070649 13658 raft_consensus.cc:729] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf8ab4ef38249adbff4d3861a539d5e, State: Initialized, Role: FOLLOWER
I20260925 16:29:58.071412 13657 consensus_queue.cc:260] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.073014 13657 ts_tablet_manager.cc:1428] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce: Time spent starting tablet: real 0.018s user 0.005s sys 0.003s
I20260925 16:29:58.073737 13658 consensus_queue.cc:260] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.074751 13661 tablet_bootstrap.cc:654] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Neither blocks nor log segments found. Creating new log.
I20260925 16:29:58.076079 13658 ts_tablet_manager.cc:1428] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Time spent starting tablet: real 0.012s user 0.004s sys 0.005s
I20260925 16:29:58.076058 13661 log.cc:824] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Log is configured to *not* fsync() on all Append() calls
I20260925 16:29:58.076668 13356 heartbeater.cc:503] Master 127.0.98.62:46223 was elected leader, sending a full tablet report...
I20260925 16:29:58.079669 13661 tablet_bootstrap.cc:492] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: No bootstrap required, opened a new log
I20260925 16:29:58.080060 13661 ts_tablet_manager.cc:1397] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 16:29:58.082406 13491 heartbeater.cc:503] Master 127.0.98.62:46223 was elected leader, sending a full tablet report...
I20260925 16:29:58.085933 13661 raft_consensus.cc:348] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.086510 13661 raft_consensus.cc:374] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:29:58.086615 13661 raft_consensus.cc:729] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38eeb6a1ba9445e9642b49845fac106, State: Initialized, Role: FOLLOWER
I20260925 16:29:58.087338 13661 consensus_queue.cc:260] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.088945 13661 ts_tablet_manager.cc:1428] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 16:29:58.089413 13631 heartbeater.cc:503] Master 127.0.98.62:46223 was elected leader, sending a full tablet report...
I20260925 16:29:58.120220 13662 raft_consensus.cc:482] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:29:58.120563 13662 raft_consensus.cc:504] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.123625 13662 leader_election.cc:288] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38eeb6a1ba9445e9642b49845fac106 (127.0.98.3:36051), 8bf8ab4ef38249adbff4d3861a539d5e (127.0.98.2:39817)
I20260925 16:29:58.129339 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.123540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45573 (local address 127.0.98.3:36051)
0925 16:29:58.123674 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:58.123679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:58.124000 (+ 321us) server_negotiation.cc:407] Connection header received
0925 16:29:58.124256 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:58.124260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:58.124320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:58.124369 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:29:58.124909 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.125695 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.127664 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.128352 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.128451 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.128549 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:58.129037 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 16:29:58.129151 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:29:58.129339 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.123175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43371 (local address 127.0.98.2:39817)
0925 16:29:58.123331 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:58.123336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:58.124120 (+ 784us) server_negotiation.cc:407] Connection header received
0925 16:29:58.124288 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:58.124292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:58.124347 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:58.124393 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:29:58.124898 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.125972 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.127661 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.128346 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.128451 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.128549 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:58.129037 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 16:29:58.129151 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:29:58.129982 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.122942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36051 (local address 127.0.98.1:45573)
0925 16:29:58.123866 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.123883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:58.124111 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:58.124537 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:58.124540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:58.124558 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:29:58.124766 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.124773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.125879 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.125883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.127503 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.127517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.128667 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.128671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.128877 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.128910 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 16:29:58.129562 (+ 652us) client_negotiation.cc:240] Negotiation successful
0925 16:29:58.129687 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":144,"spinlock_wait_cycles":32000,"thread_start_us":175,"threads_started":1}
I20260925 16:29:58.130697 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.122942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39817 (local address 127.0.98.1:43371)
0925 16:29:58.124036 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.124047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:29:58.124175 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:29:58.124537 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:29:58.124540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:29:58.124641 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:29:58.124765 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.124773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.126098 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.126101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.127503 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:29:58.127517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.128667 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.128671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:29:58.128874 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.128908 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:29:58.130436 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0925 16:29:58.130539 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":489,"spinlock_wait_cycles":37504,"thread_start_us":96,"threads_started":1}
I20260925 16:29:58.131062 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dd20a4874430ca02a6ece2e162fb6" candidate_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38eeb6a1ba9445e9642b49845fac106" is_pre_election: true
I20260925 16:29:58.131538 13585 raft_consensus.cc:2454] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0112a89312644ba390b4ef5b7f6aa7ce in term 0.
I20260925 16:29:58.131675 13443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dd20a4874430ca02a6ece2e162fb6" candidate_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" is_pre_election: true
I20260925 16:29:58.132066 13443 raft_consensus.cc:2454] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0112a89312644ba390b4ef5b7f6aa7ce in term 0.
I20260925 16:29:58.132658 13243 leader_election.cc:302] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0112a89312644ba390b4ef5b7f6aa7ce, e38eeb6a1ba9445e9642b49845fac106; no voters:
I20260925 16:29:58.132992 13662 raft_consensus.cc:2781] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:29:58.133080 13662 raft_consensus.cc:482] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:29:58.133126 13662 raft_consensus.cc:3037] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:29:58.134841 13662 raft_consensus.cc:504] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.135308 13662 leader_election.cc:288] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [CANDIDATE]: Term 1 election: Requested vote from peers e38eeb6a1ba9445e9642b49845fac106 (127.0.98.3:36051), 8bf8ab4ef38249adbff4d3861a539d5e (127.0.98.2:39817)
I20260925 16:29:58.136011 13443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dd20a4874430ca02a6ece2e162fb6" candidate_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8ab4ef38249adbff4d3861a539d5e"
I20260925 16:29:58.136183 13443 raft_consensus.cc:3037] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:29:58.136286 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dd20a4874430ca02a6ece2e162fb6" candidate_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38eeb6a1ba9445e9642b49845fac106"
I20260925 16:29:58.136447 13585 raft_consensus.cc:3037] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 16:29:58 dist-test-slave-31kt krb5kdc[12161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 16:29:58.137892 13585 raft_consensus.cc:2454] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0112a89312644ba390b4ef5b7f6aa7ce in term 1.
I20260925 16:29:58.137892 13443 raft_consensus.cc:2454] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0112a89312644ba390b4ef5b7f6aa7ce in term 1.
I20260925 16:29:58.138352 13243 leader_election.cc:302] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0112a89312644ba390b4ef5b7f6aa7ce, e38eeb6a1ba9445e9642b49845fac106; no voters:
I20260925 16:29:58.138688 13662 raft_consensus.cc:2781] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:29:58.139000 13662 raft_consensus.cc:686] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [term 1 LEADER]: Becoming Leader. State: Replica: 0112a89312644ba390b4ef5b7f6aa7ce, State: Running, Role: LEADER
I20260925 16:29:58.139515 13662 consensus_queue.cc:237] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } }
I20260925 16:29:58.144057 13154 catalog_manager.cc:5495] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0112a89312644ba390b4ef5b7f6aa7ce (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0112a89312644ba390b4ef5b7f6aa7ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36369 } health_report { overall_health: HEALTHY } } }
W20260925 16:29:58.165970 13492 tablet.cc:2369] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:29:58.179704 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.173515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.98.1:36369)
0925 16:29:58.173941 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 16:29:58.173947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:29:58.173967 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:29:58.174089 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:29:58.174092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:29:58.174182 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:29:58.174330 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:29:58.174734 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.175610 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.176871 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:29:58.177124 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:29:58.177240 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:29:58.177743 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:29:58.178504 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:29:58.178669 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:29:58.179338 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 16:29:58.179452 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260925 16:29:58.190860 13443 raft_consensus.cc:1261] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 1 FOLLOWER]: Refusing update from remote peer 0112a89312644ba390b4ef5b7f6aa7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:29:58.190860 13585 raft_consensus.cc:1261] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 1 FOLLOWER]: Refusing update from remote peer 0112a89312644ba390b4ef5b7f6aa7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:29:58.192248 13662 consensus_queue.cc:1035] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf8ab4ef38249adbff4d3861a539d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:29:58.193121 13662 consensus_queue.cc:1035] T 877dd20a4874430ca02a6ece2e162fb6 P 0112a89312644ba390b4ef5b7f6aa7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38eeb6a1ba9445e9642b49845fac106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:29:58.211243 13671 mvcc.cc:205] Tried to move back new op lower bound from 7333289157376462848 to 7333289157183709184. Current Snapshot: MvccSnapshot[applied={T|T < 7333289157376462848}]
I20260925 16:29:58.214792 13674 mvcc.cc:205] Tried to move back new op lower bound from 7333289157376462848 to 7333289157183709184. Current Snapshot: MvccSnapshot[applied={T|T < 7333289157376462848}]
I20260925 16:29:58.219277 13673 mvcc.cc:205] Tried to move back new op lower bound from 7333289157376462848 to 7333289157183709184. Current Snapshot: MvccSnapshot[applied={T|T < 7333289157376462848}]
I20260925 16:29:58.237044 13153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:29:58.237357 13153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:29:58.241472 13153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5f0a20ad8f4e72ac3734e251276c42: Sending DeleteTablet for 3 replicas of tablet 877dd20a4874430ca02a6ece2e162fb6
I20260925 16:29:58.244884 13425 tablet_service.cc:1514] Processing DeleteTablet for tablet 877dd20a4874430ca02a6ece2e162fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:29:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49966
I20260925 16:29:58.245725 13565 tablet_service.cc:1514] Processing DeleteTablet for tablet 877dd20a4874430ca02a6ece2e162fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:29:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46872
I20260925 16:29:58.247577 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 877dd20a4874430ca02a6ece2e162fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:29:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41880
I20260925 16:29:58.249805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13196
I20260925 16:29:58.250826 13681 tablet_replica.cc:330] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: stopping tablet replica
I20260925 16:29:58.251318 13681 raft_consensus.cc:2229] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:29:58.251768 13681 raft_consensus.cc:2258] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:29:58.258381 13682 tablet_replica.cc:330] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: stopping tablet replica
I20260925 16:29:58.258904 13682 raft_consensus.cc:2229] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:29:58.259387 13682 raft_consensus.cc:2258] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:29:58.261202 13682 ts_tablet_manager.cc:1905] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:29:58.261202 13681 ts_tablet_manager.cc:1905] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:29:58.265174 13681 ts_tablet_manager.cc:1918] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:29:58.265323 13681 log.cc:1196] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/877dd20a4874430ca02a6ece2e162fb6
I20260925 16:29:58.265472 13682 ts_tablet_manager.cc:1918] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:29:58.265655 13682 log.cc:1196] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/877dd20a4874430ca02a6ece2e162fb6
I20260925 16:29:58.266101 13682 ts_tablet_manager.cc:1939] T 877dd20a4874430ca02a6ece2e162fb6 P e38eeb6a1ba9445e9642b49845fac106: Deleting consensus metadata
I20260925 16:29:58.268457 13142 catalog_manager.cc:4841] TS e38eeb6a1ba9445e9642b49845fac106 (127.0.98.3:36051): tablet 877dd20a4874430ca02a6ece2e162fb6 (table test-table [id=16428ca694574948a4e29568db89bb66]) successfully deleted
W20260925 16:29:58.270393 13143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36369 (error 108)
I20260925 16:29:58.272333 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.271902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36369 (local address 127.0.0.1:41904)
0925 16:29:58.272111 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.272209 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 16:29:58.272598 13143 catalog_manager.cc:4584] TS 0112a89312644ba390b4ef5b7f6aa7ce (127.0.98.1:36369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 877dd20a4874430ca02a6ece2e162fb6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36369: connect: Connection refused (error 111)
I20260925 16:29:58.265718 13681 ts_tablet_manager.cc:1939] T 877dd20a4874430ca02a6ece2e162fb6 P 8bf8ab4ef38249adbff4d3861a539d5e: Deleting consensus metadata
I20260925 16:29:58.274421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13358
I20260925 16:29:58.284216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13494
I20260925 16:29:58.287065 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:29:58.286615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39817 (local address 127.0.0.1:49972)
0925 16:29:58.286835 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 16:29:58.286935 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 16:29:58.294363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13125
I20260925 16:29:58.302523 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 16:29:58.639559 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 16:29:58.639721 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 16:29:58.987197 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 16:29:58.987 UTC [12191] LOG: received smart shutdown request
2026-09-25 16:29:58.992 UTC [12191] LOG: background worker "logical replication launcher" (PID 12200) exited with exit code 1
2026-09-25 16:29:58.992 UTC [12195] LOG: shutting down
2026-09-25 16:29:59.212 UTC [12191] LOG: database system is shut down
2026-09-25T16:29:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58731 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:29:59 dist-test-slave-31kt krb5kdc[13693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:29:59 dist-test-slave-31kt krb5kdc[13693](info): set up 2 sockets
Sep 25 16:29:59 dist-test-slave-31kt krb5kdc[13693](info): commencing operation
krb5kdc: starting...
W20260925 16:30:00.318130 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:30:00 dist-test-slave-31kt krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:30:00.458127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43563
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43563,127.0.98.61:41629,127.0.98.60:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:00.611876 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:00.612367 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:00.612437 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:00.617378 13706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:00.617503 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:00.617537 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:00.617559 13706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:00.617581 13706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:00.624625 13706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43563,127.0.98.61:41629,127.0.98.60:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:00.626411 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:00.627868 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:30:00.640254 13706 server_base.cc:1034] running on GCE node
W20260925 16:30:00.640074 13712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:00.640089 13711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:00.640091 13714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:00.641422 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:00.643245 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:00.644497 13706 hybrid_clock.cc:648] HybridClock initialized: now 1790353800644461 us; error 63 us; skew 500 ppm
Sep 25 16:30:00 dist-test-slave-31kt krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:00.652779 13706 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:00.654268 13706 webserver.cc:460] Webserver started at http://127.0.98.62:33921/ using document root <none> and password file <none>
I20260925 16:30:00.655032 13706 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:00.655115 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:00.655377 13706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:00.658346 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "6173375b3a51410aa06094c974e8d7f1"
format_stamp: "Formatted at 2026-09-25 16:30:00 on dist-test-slave-31kt"
server_key: "fe3bcc3cc0c2402d53b3234c56099cc2"
server_key_iv: "b58d4e0b05979c8670340a246b672de5"
server_key_version: "encryptionkey@0"
I20260925 16:30:00.659085 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "6173375b3a51410aa06094c974e8d7f1"
format_stamp: "Formatted at 2026-09-25 16:30:00 on dist-test-slave-31kt"
server_key: "fe3bcc3cc0c2402d53b3234c56099cc2"
server_key_iv: "b58d4e0b05979c8670340a246b672de5"
server_key_version: "encryptionkey@0"
I20260925 16:30:00.664048 13706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:30:00.667933 13721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:00.670055 13706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:00.670252 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "6173375b3a51410aa06094c974e8d7f1"
format_stamp: "Formatted at 2026-09-25 16:30:00 on dist-test-slave-31kt"
server_key: "fe3bcc3cc0c2402d53b3234c56099cc2"
server_key_iv: "b58d4e0b05979c8670340a246b672de5"
server_key_version: "encryptionkey@0"
I20260925 16:30:00.670432 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:00.684834 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:00.690013 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:00.690274 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:00.700287 13706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43563
I20260925 16:30:00.700398 13773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43563 every 8 connection(s)
I20260925 16:30:00.701785 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:00.706591 13774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:00.708782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13706
I20260925 16:30:00.708956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:30:00.709236 392 external_mini_cluster.cc:1428] Setting key d411e616eae86a07799909667c23b6e8
I20260925 16:30:00.712014 13774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 16:30:00 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:00.734341 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.714932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43563 (local address 127.0.0.1:50856)
0925 16:30:00.715789 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.715853 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:00.716853 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:00.718238 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:00.718245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:00.718736 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:00.719124 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:00.719145 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:00.720711 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:00.720715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:00.722000 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:00.722018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:00.723437 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:00.723442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:00.723566 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:00.724349 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:00.724371 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:00.726737 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:00.729435 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:00.729457 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:00.729475 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:00.730155 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:00.730764 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:00.730768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:00.730770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:00.731036 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:00.732038 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:00.732048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:00.732313 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 16:30:00.732577 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 16:30:00.733191 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":149,"spinlock_wait_cycles":27520,"thread_start_us":184,"threads_started":1}
I20260925 16:30:00.734503 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.714932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.0.98.62:43563)
0925 16:30:00.715768 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:00.715806 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:00.716034 (+ 228us) server_negotiation.cc:407] Connection header received
0925 16:30:00.717341 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:00.717351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:00.717849 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:00.718161 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:00.719365 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:00.720476 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:00.722224 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:00.722828 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:00.723072 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:00.727069 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:00.727079 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:00.727098 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:00.727138 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:00.729189 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:00.730394 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:00.730402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:00.730404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:00.730609 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:00.731243 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:00.731246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:00.731247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:00.731859 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:00.732007 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:00.732711 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 16:30:00.732992 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":183,"threads_started":1}
I20260925 16:30:00.738085 13774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:30:00.739362 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.738781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47070)
0925 16:30:00.738981 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.739085 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 16:30:00.739645 13724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
I20260925 16:30:00.741055 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.740572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47084)
0925 16:30:00.740837 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.740942 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260925 16:30:00.741335 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 16:30:00.745193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41629
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43563,127.0.98.61:41629,127.0.98.60:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:00.790711 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } attempt: 1
I20260925 16:30:00.792076 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.791585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47096)
0925 16:30:00.791837 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.791950 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 16:30:00.793212 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.792910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47112)
0925 16:30:00.793046 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.793125 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 16:30:00.793434 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
I20260925 16:30:00.861851 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } attempt: 2
I20260925 16:30:00.863545 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.862973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47126)
0925 16:30:00.863214 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.863418 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 16:30:00.864678 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.864318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47130)
0925 16:30:00.864489 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.864583 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 16:30:00.864948 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
W20260925 16:30:00.893003 13781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:00.893363 13781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:00.893437 13781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:00.898180 13781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:00.898283 13781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:00.898303 13781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:00.898320 13781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:00.898335 13781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:00.906349 13781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43563,127.0.98.61:41629,127.0.98.60:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:00.907629 13781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:00.908809 13781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:00.920472 13787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:00.920490 13786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:00.920572 13781 server_base.cc:1034] running on GCE node
W20260925 16:30:00.920472 13789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:00.921743 13781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:00.923245 13781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:00.926644 13781 hybrid_clock.cc:648] HybridClock initialized: now 1790353800926578 us; error 88 us; skew 500 ppm
Sep 25 16:30:00 dist-test-slave-31kt krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:00.930198 13781 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 16:30:00.931473 13781 webserver.cc:460] Webserver started at http://127.0.98.61:42857/ using document root <none> and password file <none>
I20260925 16:30:00.932126 13781 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:00.932199 13781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:00.932415 13781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:00.934831 13781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data/instance:
uuid: "e0c679a6b8cf464c9006059ac641f520"
format_stamp: "Formatted at 2026-09-25 16:30:00 on dist-test-slave-31kt"
server_key: "dc9884350e2310840296ea75b001c145"
server_key_iv: "1c9de8fe5bd44076a85146a76ea8daa5"
server_key_version: "encryptionkey@0"
I20260925 16:30:00.935714 13781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal/instance:
uuid: "e0c679a6b8cf464c9006059ac641f520"
format_stamp: "Formatted at 2026-09-25 16:30:00 on dist-test-slave-31kt"
server_key: "dc9884350e2310840296ea75b001c145"
server_key_iv: "1c9de8fe5bd44076a85146a76ea8daa5"
server_key_version: "encryptionkey@0"
I20260925 16:30:00.941438 13781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:30:00.945519 13796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:00.947495 13781 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:00.947736 13781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal
uuid: "e0c679a6b8cf464c9006059ac641f520"
format_stamp: "Formatted at 2026-09-25 16:30:00 on dist-test-slave-31kt"
server_key: "dc9884350e2310840296ea75b001c145"
server_key_iv: "1c9de8fe5bd44076a85146a76ea8daa5"
server_key_version: "encryptionkey@0"
I20260925 16:30:00.947919 13781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:00.956184 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } attempt: 3
I20260925 16:30:00.957513 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.956905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47146)
0925 16:30:00.957286 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.957384 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 16:30:00.958680 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:00.958400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47150)
0925 16:30:00.958522 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:00.958584 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 16:30:00.958885 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41629: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41629: connect: Connection refused (error 111)
I20260925 16:30:01.011570 13781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:01.015753 13781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:01.015967 13781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:01.027208 13781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41629
I20260925 16:30:01.027338 13848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41629 every 8 connection(s)
I20260925 16:30:01.028415 13781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:30:01.032646 13849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:01.037096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13781
I20260925 16:30:01.037292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-1/wal/instance
I20260925 16:30:01.037612 392 external_mini_cluster.cc:1428] Setting key f6b2ae1f24093aae28bcc05f9a2beb6f
I20260925 16:30:01.038967 13849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:01.065490 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.042626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.98.62:43563)
0925 16:30:01.042798 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.042802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.044893 (+ 2091us) server_negotiation.cc:407] Connection header received
0925 16:30:01.046388 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.046394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.046486 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.046573 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.048459 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.049072 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.051213 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.051710 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.051847 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.056833 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.056838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.056841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.056886 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.061014 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.062206 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.062211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.062212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.062330 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.063375 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.063380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.063381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.063706 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.063818 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.065094 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.065272 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 16:30:01.066761 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.043486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43563 (local address 127.0.0.1:50864)
0925 16:30:01.044646 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.044719 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.046061 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.046867 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.046879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.047405 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.048171 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.048194 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.049395 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.049399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.050692 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.050711 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.052004 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.052008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.052315 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.053318 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.053348 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.056394 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.061295 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.061310 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.061346 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.062010 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.062450 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.062454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.062456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.062737 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.063869 (+ 1132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.063882 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.064448 (+ 566us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.064991 (+ 543us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.065581 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":355,"threads_started":1}
I20260925 16:30:01.069348 13849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 16:30:01.079317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45309
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43563,127.0.98.61:41629,127.0.98.60:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:01.082363 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.070122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47154)
0925 16:30:01.070360 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.070379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.070517 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.071250 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.071253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.071470 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.071559 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.071566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.072655 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.072658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.073228 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.073237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.073924 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.073927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.074029 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.074581 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.074594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.076372 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.079531 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.079536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.079539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.079832 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.080449 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.080452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.080454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.080580 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.081513 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.081517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.081626 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.081967 (+ 341us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.082174 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 16:30:01.083144 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.070287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.98.61:41629)
0925 16:30:01.070757 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.070768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.070813 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:30:01.070872 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.070877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.071101 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.071259 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.071672 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.072546 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.073340 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.073810 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.073916 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.076661 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.076683 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.076723 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.076761 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.079309 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.080084 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.080093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.080095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.080271 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.080793 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.080797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.080798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.081321 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.081519 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.082200 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.082414 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":208,"threads_started":1}
I20260925 16:30:01.086287 13849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:30:01.087853 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.087286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43890)
0925 16:30:01.087540 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.087713 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260925 16:30:01.088330 13798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.090131 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.089440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43906)
0925 16:30:01.089629 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.089733 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 16:30:01.090441 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.102108 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } attempt: 4
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:30:01.114411 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.102910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47166)
0925 16:30:01.103145 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.103165 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.103308 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.103752 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.103756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.104029 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.104181 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.104189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.105529 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.105533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.106455 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.106476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.107403 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.107408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.107540 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.108234 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.108248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.110136 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.112547 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.112551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.112553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.112841 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.113158 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.113162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.113164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.113250 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.113696 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.113700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.113869 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.113969 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.114184 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 16:30:01.114411 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.103078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.0.98.61:41629)
0925 16:30:01.103295 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.103301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.103330 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:30:01.103414 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.103492 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.103578 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.103696 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.104335 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.105363 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.106647 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.107222 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.107364 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.110395 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.110399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.110401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.110442 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.112333 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.112963 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.112966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.112968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.113064 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.113349 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.113353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.113355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.113600 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.113697 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.114044 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.114236 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 16:30:01.115765 13774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:30:01.117018 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.116582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43910)
0925 16:30:01.116791 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.116898 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 16:30:01.118148 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.117833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43924)
0925 16:30:01.117964 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.118036 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 16:30:01.118350 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.139843 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 1
I20260925 16:30:01.141127 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.140709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43928)
0925 16:30:01.140917 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.141005 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 16:30:01.142251 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.141778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43940)
0925 16:30:01.142002 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.142110 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 16:30:01.142488 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.177556 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 1
I20260925 16:30:01.178750 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.178284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43946)
0925 16:30:01.178514 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.178619 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 16:30:01.179730 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.179453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43950)
0925 16:30:01.179578 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.179652 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:30:01.179930 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.210682 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 2
I20260925 16:30:01.211928 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.211455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43960)
0925 16:30:01.211704 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.211801 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 16:30:01.212900 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.212618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43970)
0925 16:30:01.212751 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.212819 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:30:01.213099 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
W20260925 16:30:01.224951 13855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:01.225404 13855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:01.225454 13855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:01.231400 13855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:01.231515 13855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:01.231537 13855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:01.231555 13855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:01.231572 13855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:01.239864 13855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43563,127.0.98.61:41629,127.0.98.60:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:01.241351 13855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:01.242775 13855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:30:01.247165 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 2
I20260925 16:30:01.248440 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.247995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43976)
0925 16:30:01.248205 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.248318 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 16:30:01.249358 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.249101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43984)
0925 16:30:01.249210 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.249276 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 16:30:01.249543 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.253109 13855 server_base.cc:1034] running on GCE node
W20260925 16:30:01.253069 13864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:01.253019 13861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:01.253022 13862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:01.254185 13855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:01.255466 13855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:01.256680 13855 hybrid_clock.cc:648] HybridClock initialized: now 1790353801256645 us; error 62 us; skew 500 ppm
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:01.260600 13855 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 16:30:01.261745 13855 webserver.cc:460] Webserver started at http://127.0.98.60:36051/ using document root <none> and password file <none>
I20260925 16:30:01.262398 13855 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:01.262589 13855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:01.263046 13855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:01.265851 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data/instance:
uuid: "0dd1f1e3a7d7482481082c60cc9d83e9"
format_stamp: "Formatted at 2026-09-25 16:30:01 on dist-test-slave-31kt"
server_key: "24a1320bdacb9d75144b7a1e65f7d8c3"
server_key_iv: "cbd470b63089b51fcc26e158643d3470"
server_key_version: "encryptionkey@0"
I20260925 16:30:01.266498 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal/instance:
uuid: "0dd1f1e3a7d7482481082c60cc9d83e9"
format_stamp: "Formatted at 2026-09-25 16:30:01 on dist-test-slave-31kt"
server_key: "24a1320bdacb9d75144b7a1e65f7d8c3"
server_key_iv: "cbd470b63089b51fcc26e158643d3470"
server_key_version: "encryptionkey@0"
I20260925 16:30:01.271052 13855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 16:30:01.274284 13871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:01.276160 13855 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260925 16:30:01.276379 13855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal
uuid: "0dd1f1e3a7d7482481082c60cc9d83e9"
format_stamp: "Formatted at 2026-09-25 16:30:01 on dist-test-slave-31kt"
server_key: "24a1320bdacb9d75144b7a1e65f7d8c3"
server_key_iv: "cbd470b63089b51fcc26e158643d3470"
server_key_version: "encryptionkey@0"
I20260925 16:30:01.276716 13855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:01.295468 13855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:01.300197 13855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:01.300398 13855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:01.304360 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 3
I20260925 16:30:01.305686 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.305203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:43986)
0925 16:30:01.305436 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.305557 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 16:30:01.307116 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.306741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:44000)
0925 16:30:01.306938 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.307002 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260925 16:30:01.307308 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45309: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45309: connect: Connection refused (error 111)
I20260925 16:30:01.309548 13855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45309
I20260925 16:30:01.309659 13923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45309 every 8 connection(s)
I20260925 16:30:01.311005 13855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:30:01.315613 13924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:01.320744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 13855
I20260925 16:30:01.320904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790353612433705-392-0/minicluster-data/master-2/wal/instance
I20260925 16:30:01.321530 392 external_mini_cluster.cc:1428] Setting key 0e8b1821f0e1b75f3e6150344fddf2e9
I20260925 16:30:01.321581 13924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:01.337018 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.324666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.0.98.62:43563)
0925 16:30:01.324843 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.324847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.325247 (+ 400us) server_negotiation.cc:407] Connection header received
0925 16:30:01.325716 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.325719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.325817 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.325899 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.326959 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.327510 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.328662 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.328889 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.328984 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.332504 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.332507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.332509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.332542 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.334088 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.334804 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.334809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.334810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.334926 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.335573 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.335577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.335579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.335947 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.336068 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.336608 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.336799 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 16:30:01.338238 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.324474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43563 (local address 127.0.0.1:50868)
0925 16:30:01.325035 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.325132 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.325504 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.325994 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.326000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.326459 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.326801 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.326814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.327630 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.327634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.328535 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.328546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.329059 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.329064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.329214 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.330022 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.330041 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.332309 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.334223 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.334235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.334251 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.334665 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.335089 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.335093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.335096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.335355 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.336091 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.336098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.336295 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.336550 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.336986 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":48,"spinlock_wait_cycles":896,"thread_start_us":164,"threads_started":1}
I20260925 16:30:01.338267 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.323346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.0.98.60:45309)
0925 16:30:01.323906 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.323929 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.323976 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:30:01.325106 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.325133 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.325508 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.325818 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.326770 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.327738 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.328603 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.329035 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.329214 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.331896 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.331904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.331921 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.331949 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.334220 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.334775 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.334783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.334785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.334962 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.335573 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.335577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.335579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.336279 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.336462 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.336776 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.337298 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":323,"server-negotiator.queue_time_us":385,"thread_start_us":151,"threads_started":1}
I20260925 16:30:01.339967 13924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:01.349735 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 3
I20260925 16:30:01.351034 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.340817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.0.98.61:41629)
0925 16:30:01.341001 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.341006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.341023 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:30:01.341069 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.341072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.341125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.341208 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.342003 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.342585 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.343929 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.344235 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.344471 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.347126 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.347130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.347134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.347166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.348845 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.349421 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.349425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.349426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.349545 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.349897 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.349900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.349902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.350168 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.350264 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.350649 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.350841 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:30:01.351456 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.341339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.0.98.62:43563)
0925 16:30:01.341480 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.341484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.341500 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:30:01.341604 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.341607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.341661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.341785 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.342479 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.343068 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.344225 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.344693 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.344797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.347612 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.347617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.347620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.347653 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.349043 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.349604 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.349607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.349609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.349702 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.350143 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.350148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.350150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.350444 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.350715 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.351073 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.351272 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 16:30:01.351948 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.340694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41629 (local address 127.0.0.1:47182)
0925 16:30:01.340837 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.340855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.340948 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.341463 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.341467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.341702 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.341871 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.341880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.342807 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.342819 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.343743 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.343757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.344468 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.344472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.344556 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.345210 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.345226 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.346904 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.349000 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.349004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.349006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.349262 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.349666 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.349668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.349670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.349801 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.350295 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.350299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.350437 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.351561 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.351748 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 16:30:01.352810 13924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:30:01.360553 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.350417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:44018)
0925 16:30:01.350664 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.350681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.350755 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.351144 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.351148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.351374 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.351506 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.351525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.352595 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.352600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.353376 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.353386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.354606 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.354610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.354708 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.355239 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.355253 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.356874 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.358756 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.358760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.358763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.358984 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.359295 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.359299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.359302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.359434 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.359947 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.359951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.360080 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.360163 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.360327 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 16:30:01.361413 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.350469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.98.60:45309)
0925 16:30:01.350923 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.350927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.350939 (+ 12us) server_negotiation.cc:407] Connection header received
0925 16:30:01.350972 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.350976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.351017 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.351629 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.351713 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.352396 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.353494 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.354487 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.354590 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.357063 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.357067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.357069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.357099 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.358606 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.359085 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.359088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.359089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.359170 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.359537 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.359541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.359543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.359812 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.361069 (+ 1257us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.361147 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.361274 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":373}
I20260925 16:30:01.363407 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.353366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:44030)
0925 16:30:01.353476 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.353489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.353559 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.354195 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.354198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.354370 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.354485 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.354491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.355693 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.355697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.356263 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.356272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.356795 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.356799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.356903 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.357358 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.357373 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.359041 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.361167 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.361171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.361174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.361435 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.362046 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.362049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.362052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.362153 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.362816 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.362820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.362914 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.363032 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.363201 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":17}
I20260925 16:30:01.363561 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.353578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.0.98.60:45309)
0925 16:30:01.353978 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.353988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.353999 (+ 11us) server_negotiation.cc:407] Connection header received
0925 16:30:01.354037 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.354040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.354079 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.354580 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.354725 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.355559 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.356371 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.356693 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.356785 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.359205 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.359209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.359213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.359235 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.361020 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.361581 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.361585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.361587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.361926 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.362298 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.362303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.362305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.362671 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.362780 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.363154 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.363318 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":336,"thread_start_us":190,"threads_started":1}
I20260925 16:30:01.364713 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.353561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.0.98.61:41629)
0925 16:30:01.353707 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.353711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.353724 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:30:01.353815 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.353818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.353870 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.353964 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.354792 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.355559 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.356697 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.356997 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.357069 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.360001 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.360004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.360006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.360042 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.361958 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.362677 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.362682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.362684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.362803 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.363159 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.363164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.363166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.363444 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.363713 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.364448 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.364591 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 16:30:01.366189 13774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: Bootstrap starting.
I20260925 16:30:01.368599 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: Bootstrap starting.
I20260925 16:30:01.369985 13774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:01.370421 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:30:01.371372 13774 log.cc:824] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:01.372304 13924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:01.373481 13924 log.cc:824] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:01.375804 13774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: No bootstrap required, opened a new log
I20260925 16:30:01.377724 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: No bootstrap required, opened a new log
I20260925 16:30:01.382483 13774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.382846 13774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:01.382928 13774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173375b3a51410aa06094c974e8d7f1, State: Initialized, Role: FOLLOWER
I20260925 16:30:01.383584 13774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.383590 13924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.384253 13924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:01.384385 13924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd1f1e3a7d7482481082c60cc9d83e9, State: Initialized, Role: FOLLOWER
I20260925 16:30:01.384821 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.373340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.0.98.61:41629)
0925 16:30:01.373486 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.373491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.373510 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:30:01.373859 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.373865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.373938 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.374003 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.375420 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.376269 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.377699 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.378089 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.378182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.380395 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.380400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.380403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.380444 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.382501 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.383143 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.383148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.383150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.383246 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.383537 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.383553 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.383554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.383846 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.384016 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.384455 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.384634 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:30:01.385193 13924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.385480 13774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:01.387251 13924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:01.388155 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.372897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.0.98.62:43563)
0925 16:30:01.373913 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.373918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.373934 (+ 16us) server_negotiation.cc:407] Connection header received
0925 16:30:01.373985 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.373989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.374051 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.374145 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.375244 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.376017 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.380167 (+ 4150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.380569 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.380667 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.383161 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.383166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.383170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.383208 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.384902 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.385495 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.385499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.385501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.385607 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.385962 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.385966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.385968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.386218 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.386350 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.387821 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.387979 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912}
I20260925 16:30:01.391299 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
W20260925 16:30:01.395853 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 16:30:01.391729 13706 master_runner.cc:186] Error getting master registration for 127.0.98.61:41629: OK, instance_id { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" instance_seqno: 1790353801024493 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:30:01.396445 13706 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 16:30:01.396121 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:30:01.391742 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:30:01.397652 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.373457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.0.98.60:45309)
0925 16:30:01.374901 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.374907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.386287 (+ 11380us) server_negotiation.cc:407] Connection header received
0925 16:30:01.386475 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.386478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.386552 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.386606 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.387444 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.388287 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.389493 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.389888 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.389970 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.392998 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.393004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.393008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.393048 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.394874 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.395484 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.395488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.395490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.395578 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.395870 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.395874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.395876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.396338 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.396574 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.397247 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.397448 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1336}
I20260925 16:30:01.398602 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.398797 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:30:01.395853 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:30:01.398983 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:30:01.403424 13855 master_runner.cc:186] Error getting master registration for 127.0.98.61:41629: OK, instance_id { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" instance_seqno: 1790353801024493 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:30:01.408860 13855 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 16:30:01.442950 13727 tablet.cc:2369] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:01.450495 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } attempt: 4
Sep 25 16:30:01 dist-test-slave-31kt krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:30:01.463562 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.451325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45309 (local address 127.0.0.1:44040)
0925 16:30:01.451622 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:01.451639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:01.451790 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:01.452195 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:01.452198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:01.452461 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:01.452647 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.452655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.453927 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.453932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.454878 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:01.454890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.455950 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.455954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:01.456051 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.457126 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:01.457144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:01.459129 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:01.461447 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.461453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.461456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.461892 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.462257 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:01.462261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:01.462263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:01.462366 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:01.462856 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:01.462859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:01.463016 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 16:30:01.463134 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:30:01.463343 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260925 16:30:01.463693 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:01.451473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.98.60:45309)
0925 16:30:01.451636 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:01.451641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:01.451732 (+ 91us) server_negotiation.cc:407] Connection header received
0925 16:30:01.451922 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:01.451925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:01.451996 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:01.452133 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:01.452801 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.453714 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.455070 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:01.455461 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:01.455570 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:01.459370 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:01.459375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:01.459379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:01.459418 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:01.461265 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.462035 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.462039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.462041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.462156 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:01.462475 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:01.462478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:01.462480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:01.462736 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:01.462860 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:01.463203 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 16:30:01.463361 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:30:01.469182 13849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: Bootstrap starting.
I20260925 16:30:01.473368 13849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:01.474661 13849 log.cc:824] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:01.478638 13849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: No bootstrap required, opened a new log
I20260925 16:30:01.484066 13849 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.484439 13849 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:01.484488 13849 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c679a6b8cf464c9006059ac641f520, State: Initialized, Role: FOLLOWER
I20260925 16:30:01.485131 13849 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.487533 13849 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:01.488538 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.488850 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:30:01.495399 13977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:30:01.495623 13977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:30:01.497999 13781 master_runner.cc:263] No leader master found from master e0c679a6b8cf464c9006059ac641f520
I20260925 16:30:01.498127 13781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 16:30:01.518162 13802 tablet.cc:2369] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:01.536796 13939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:01.537163 13939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.538740 13939 leader_election.cc:288] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0c679a6b8cf464c9006059ac641f520 (127.0.98.61:41629), 0dd1f1e3a7d7482481082c60cc9d83e9 (127.0.98.60:45309)
I20260925 16:30:01.540263 13823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6173375b3a51410aa06094c974e8d7f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c679a6b8cf464c9006059ac641f520" is_pre_election: true
I20260925 16:30:01.540771 13823 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173375b3a51410aa06094c974e8d7f1 in term 0.
I20260925 16:30:01.541345 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173375b3a51410aa06094c974e8d7f1, e0c679a6b8cf464c9006059ac641f520; no voters:
I20260925 16:30:01.542081 13898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6173375b3a51410aa06094c974e8d7f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" is_pre_election: true
I20260925 16:30:01.542553 13898 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6173375b3a51410aa06094c974e8d7f1 in term 0.
I20260925 16:30:01.543375 13939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:30:01.543504 13939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:30:01.543568 13939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:01.545562 13939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.547200 13823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6173375b3a51410aa06094c974e8d7f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c679a6b8cf464c9006059ac641f520"
I20260925 16:30:01.547456 13823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:01.549612 13823 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173375b3a51410aa06094c974e8d7f1 in term 1.
W20260925 16:30:01.552270 13877 tablet.cc:2369] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:01.554636 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173375b3a51410aa06094c974e8d7f1, e0c679a6b8cf464c9006059ac641f520; no voters:
I20260925 16:30:01.557802 13898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6173375b3a51410aa06094c974e8d7f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9"
I20260925 16:30:01.558076 13898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:01.560138 13898 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6173375b3a51410aa06094c974e8d7f1 in term 1.
I20260925 16:30:01.560853 13939 leader_election.cc:288] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [CANDIDATE]: Term 1 election: Requested vote from peers e0c679a6b8cf464c9006059ac641f520 (127.0.98.61:41629), 0dd1f1e3a7d7482481082c60cc9d83e9 (127.0.98.60:45309)
I20260925 16:30:01.561153 13939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:01.561281 13939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [term 1 LEADER]: Becoming Leader. State: Replica: 6173375b3a51410aa06094c974e8d7f1, State: Running, Role: LEADER
I20260925 16:30:01.561854 13939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } }
I20260925 16:30:01.564781 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6173375b3a51410aa06094c974e8d7f1. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.564993 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:01.565551 13981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:01.567034 13981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:01.582309 13898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [term 1 FOLLOWER]: Refusing update from remote peer 6173375b3a51410aa06094c974e8d7f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:01.583416 13979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:01.589980 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6173375b3a51410aa06094c974e8d7f1. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.590222 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:30:01.592162 13823 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [term 1 FOLLOWER]: Refusing update from remote peer 6173375b3a51410aa06094c974e8d7f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:01.596824 13979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:01.603091 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6173375b3a51410aa06094c974e8d7f1. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.603351 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:30:01.611853 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.612103 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:01.612214 13982 mvcc.cc:205] Tried to move back new op lower bound from 7333289171267743744 to 7333289171202342912. Current Snapshot: MvccSnapshot[applied={T|T < 7333289171267743744}]
I20260925 16:30:01.612820 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.612968 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:01.614286 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.614449 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:30:01.615161 13983 mvcc.cc:205] Tried to move back new op lower bound from 7333289171267743744 to 7333289171202342912. Current Snapshot: MvccSnapshot[applied={T|T < 7333289171267743744}]
I20260925 16:30:01.616935 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6173375b3a51410aa06094c974e8d7f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6173375b3a51410aa06094c974e8d7f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43563 } } peers { permanent_uuid: "e0c679a6b8cf464c9006059ac641f520" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41629 } } peers { permanent_uuid: "0dd1f1e3a7d7482481082c60cc9d83e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45309 } } }
I20260925 16:30:01.617087 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:30:01.618180 13985 mvcc.cc:205] Tried to move back new op lower bound from 7333289171267743744 to 7333289171202342912. Current Snapshot: MvccSnapshot[applied={T|T < 7333289171267743744}]
I20260925 16:30:01.620937 13981 catalog_manager.cc:1347] Generated new cluster ID: ee62a58b4a54473cba3d92a58e4beb6b
I20260925 16:30:01.621037 13981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:30:01.657153 13981 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:30:01.658524 13981 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:30:01.667227 13981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6173375b3a51410aa06094c974e8d7f1: Generated new TSK 0
I20260925 16:30:01.668359 13981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:30:02.400791 13962 catalog_manager.cc:1259] Loaded cluster ID: ee62a58b4a54473cba3d92a58e4beb6b
I20260925 16:30:02.400915 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: loading cluster ID for follower catalog manager: success
I20260925 16:30:02.402997 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: acquiring CA information for follower catalog manager: success
I20260925 16:30:02.404603 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0dd1f1e3a7d7482481082c60cc9d83e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:30:02.497292 13977 catalog_manager.cc:1259] Loaded cluster ID: ee62a58b4a54473cba3d92a58e4beb6b
I20260925 16:30:02.497406 13977 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: loading cluster ID for follower catalog manager: success
I20260925 16:30:02.499639 13977 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: acquiring CA information for follower catalog manager: success
I20260925 16:30:02.501217 13977 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0c679a6b8cf464c9006059ac641f520: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:30:02.749979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13706
I20260925 16:30:02.758159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13781
I20260925 16:30:02.765112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 13855
2026-09-25T16:30:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3441 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:30:02 dist-test-slave-31kt krb5kdc[13997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:30:02 dist-test-slave-31kt krb5kdc[13997](info): set up 2 sockets
Sep 25 16:30:02 dist-test-slave-31kt krb5kdc[13997](info): commencing operation
krb5kdc: starting...
W20260925 16:30:03.746441 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:30:03 dist-test-slave-31kt krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:30:03.885188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:04.016691 14010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:04.017035 14010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:04.017096 14010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:04.021534 14010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:04.021639 14010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:04.021667 14010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:04.021687 14010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:04.021708 14010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:04.029349 14010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:04.030691 14010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:04.031832 14010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:30:04.042754 14010 server_base.cc:1034] running on GCE node
W20260925 16:30:04.042980 14015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:04.042968 14018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:04.042985 14016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:04.044314 14010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:04.045979 14010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:04.047166 14010 hybrid_clock.cc:648] HybridClock initialized: now 1790353804047151 us; error 46 us; skew 500 ppm
Sep 25 16:30:04 dist-test-slave-31kt krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:04.050727 14010 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:04.052073 14010 webserver.cc:460] Webserver started at http://127.0.98.62:39153/ using document root <none> and password file <none>
I20260925 16:30:04.052820 14010 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:04.052894 14010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:04.053140 14010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:04.055946 14010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "7aa9b3fbf6824558b53473ba7ea98e3f"
format_stamp: "Formatted at 2026-09-25 16:30:04 on dist-test-slave-31kt"
server_key: "ecb75c6f271083398ca5551dcaa12d6e"
server_key_iv: "02c36664a57c78530aa412146ff16fc9"
server_key_version: "encryptionkey@0"
I20260925 16:30:04.056651 14010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aa9b3fbf6824558b53473ba7ea98e3f"
format_stamp: "Formatted at 2026-09-25 16:30:04 on dist-test-slave-31kt"
server_key: "ecb75c6f271083398ca5551dcaa12d6e"
server_key_iv: "02c36664a57c78530aa412146ff16fc9"
server_key_version: "encryptionkey@0"
I20260925 16:30:04.061663 14010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:30:04.065030 14025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:04.067143 14010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260925 16:30:04.067332 14010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "7aa9b3fbf6824558b53473ba7ea98e3f"
format_stamp: "Formatted at 2026-09-25 16:30:04 on dist-test-slave-31kt"
server_key: "ecb75c6f271083398ca5551dcaa12d6e"
server_key_iv: "02c36664a57c78530aa412146ff16fc9"
server_key_version: "encryptionkey@0"
I20260925 16:30:04.067730 14010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:04.085340 14010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:04.089529 14010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:04.089780 14010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:04.099342 14010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36507
I20260925 16:30:04.099447 14077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36507 every 8 connection(s)
I20260925 16:30:04.100725 14010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:04.104641 14078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:04.104735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14010
I20260925 16:30:04.104851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:30:04.105225 392 external_mini_cluster.cc:1428] Setting key c69d76450d3aa913a68f7f37e08b0744
I20260925 16:30:04.113161 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f: Bootstrap starting.
I20260925 16:30:04.116256 14078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f: Neither blocks nor log segments found. Creating new log.
Sep 25 16:30:04 dist-test-slave-31kt krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:04.117107 14078 log.cc:824] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:04.119406 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f: No bootstrap required, opened a new log
I20260925 16:30:04.123457 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:04.107042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.0.98.62:36507)
0925 16:30:04.107621 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:04.107641 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:04.107694 (+ 53us) server_negotiation.cc:407] Connection header received
0925 16:30:04.108583 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:04.108607 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:04.109095 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:04.109511 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:04.110651 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:04.111919 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:04.113229 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:04.114102 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:04.114279 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:04.117613 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:04.117622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:04.117646 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:04.117695 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:04.119774 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:04.120340 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:04.120347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:04.120354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:04.120503 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:04.120779 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:04.120782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:04.120783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:04.121376 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:04.121613 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:04.121959 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 16:30:04.122361 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":180,"threads_started":1}
I20260925 16:30:04.123730 14078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36507 } }
I20260925 16:30:04.124042 14078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:04.124097 14078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa9b3fbf6824558b53473ba7ea98e3f, State: Initialized, Role: FOLLOWER
I20260925 16:30:04.124653 14078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36507 } }
I20260925 16:30:04.124811 14078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:30:04.124884 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:30:04.124987 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:04.126170 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36507 } }
I20260925 16:30:04.126572 14078 leader_election.cc:302] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aa9b3fbf6824558b53473ba7ea98e3f; no voters:
I20260925 16:30:04.126936 14078 leader_election.cc:288] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:30:04.127161 14083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:04.127331 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:30:04.127471 14083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [term 1 LEADER]: Becoming Leader. State: Replica: 7aa9b3fbf6824558b53473ba7ea98e3f, State: Running, Role: LEADER
I20260925 16:30:04.127908 14083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36507 } }
I20260925 16:30:04.128855 14078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:04.129470 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aa9b3fbf6824558b53473ba7ea98e3f. Latest consensus state: current_term: 1 leader_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36507 } } }
I20260925 16:30:04.129776 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:04.130288 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa9b3fbf6824558b53473ba7ea98e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36507 } } }
I20260925 16:30:04.130486 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f [sys.catalog]: This master's current role is: LEADER
W20260925 16:30:04.135526 14098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:30:04.135658 14098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:30:04.135757 14091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:04.136894 14091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:04.144801 14091 catalog_manager.cc:1347] Generated new cluster ID: 2ab56adef92d48fb8dd96c07a139a782
I20260925 16:30:04.144897 14091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:30:04.156211 14091 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:30:04.157359 14091 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:30:04.162807 14091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aa9b3fbf6824558b53473ba7ea98e3f: Generated new TSK 0
I20260925 16:30:04.163990 14091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:30:04.184911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14010
2026-09-25T16:30:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1417 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:30:04 dist-test-slave-31kt krb5kdc[14105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:30:04 dist-test-slave-31kt krb5kdc[14105](info): set up 2 sockets
Sep 25 16:30:04 dist-test-slave-31kt krb5kdc[14105](info): commencing operation
krb5kdc: starting...
W20260925 16:30:05.174366 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:30:05 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:30:05.317651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:05.454378 14118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:05.454824 14118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:05.454906 14118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:05.459666 14118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:05.459812 14118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:05.459841 14118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:05.459867 14118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:05.459887 14118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:05.466506 14118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:05.468111 14118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:05.469463 14118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:05.480316 14126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:05.480469 14118 server_base.cc:1034] running on GCE node
W20260925 16:30:05.480314 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:05.480333 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:05.481505 14118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:05.483117 14118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:05.484396 14118 hybrid_clock.cc:648] HybridClock initialized: now 1790353805484375 us; error 56 us; skew 500 ppm
Sep 25 16:30:05 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:05.488109 14118 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:05.489459 14118 webserver.cc:460] Webserver started at http://127.0.98.62:32791/ using document root <none> and password file <none>
I20260925 16:30:05.490242 14118 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:05.490338 14118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:05.490625 14118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:05.493546 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "51fc0e88189044a0b69a2ab7ae766cac"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "19371018548ef46a98dcd5ae96602859"
server_key_iv: "22564d5419d62d02be2d8534bd3ef751"
server_key_version: "encryptionkey@0"
I20260925 16:30:05.494369 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "51fc0e88189044a0b69a2ab7ae766cac"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "19371018548ef46a98dcd5ae96602859"
server_key_iv: "22564d5419d62d02be2d8534bd3ef751"
server_key_version: "encryptionkey@0"
I20260925 16:30:05.499643 14118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:30:05.503278 14133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:05.505081 14118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:30:05.505270 14118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "51fc0e88189044a0b69a2ab7ae766cac"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "19371018548ef46a98dcd5ae96602859"
server_key_iv: "22564d5419d62d02be2d8534bd3ef751"
server_key_version: "encryptionkey@0"
I20260925 16:30:05.505429 14118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:05.533504 14118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:05.538120 14118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:05.538375 14118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:05.547948 14118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45435
I20260925 16:30:05.548337 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45435 every 8 connection(s)
I20260925 16:30:05.549557 14118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:05.553903 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:05.557148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14118
I20260925 16:30:05.557300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:30:05.557621 392 external_mini_cluster.cc:1428] Setting key 331d3a327ea4de40b2f6ff84bc4a0273
I20260925 16:30:05.561916 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap starting.
I20260925 16:30:05.565663 14186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:05.566960 14186 log.cc:824] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:30:05 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:05.570328 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: No bootstrap required, opened a new log
I20260925 16:30:05.575341 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:05.559613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.98.62:45435)
0925 16:30:05.560145 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:05.560166 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:05.560213 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:30:05.561016 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:05.561039 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:05.561474 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:05.561887 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:05.562944 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:05.564128 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:05.565424 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:05.566078 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:05.566217 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:05.569508 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:05.569515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:05.569535 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:05.569575 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:05.571652 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:05.572247 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:05.572254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:05.572260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:05.572427 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:05.572697 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:05.572700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:05.572701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:05.573252 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:05.573446 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:05.573810 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 16:30:05.574247 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":142,"threads_started":1}
I20260925 16:30:05.575491 14186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:05.575850 14186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:05.575909 14186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fc0e88189044a0b69a2ab7ae766cac, State: Initialized, Role: FOLLOWER
I20260925 16:30:05.576448 14186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:05.576613 14186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:30:05.576680 14186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:30:05.576859 14186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:05.578563 14186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:05.579030 14186 leader_election.cc:302] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fc0e88189044a0b69a2ab7ae766cac; no voters:
I20260925 16:30:05.579432 14186 leader_election.cc:288] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:30:05.579553 14191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:05.579807 14191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 LEADER]: Becoming Leader. State: Replica: 51fc0e88189044a0b69a2ab7ae766cac, State: Running, Role: LEADER
I20260925 16:30:05.580225 14191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:05.581007 14186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:05.582347 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51fc0e88189044a0b69a2ab7ae766cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260925 16:30:05.582351 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fc0e88189044a0b69a2ab7ae766cac. Latest consensus state: current_term: 1 leader_uuid: "51fc0e88189044a0b69a2ab7ae766cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260925 16:30:05.582554 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:05.582624 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:05.583086 14200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:05.587291 14200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:05.595386 14200 catalog_manager.cc:1347] Generated new cluster ID: ff3f731bc8ba42ceaa1f349f8afd1096
I20260925 16:30:05.595532 14200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:30:05.612108 14200 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:30:05.613252 14200 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:30:05.623797 14200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Generated new TSK 0
I20260925 16:30:05.625191 14200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:30:05.663110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45435
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:05.806610 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:05.806937 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:05.806988 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:05.812229 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:05.812314 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:05.812379 14212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:30:05.819064 14212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:05.820703 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:05.822125 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:05.834965 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:05.835026 14220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:05.835180 14212 server_base.cc:1034] running on GCE node
W20260925 16:30:05.835527 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:05.836126 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:05.837119 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:05.838366 14212 hybrid_clock.cc:648] HybridClock initialized: now 1790353805838345 us; error 49 us; skew 500 ppm
Sep 25 16:30:05 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:05.849875 14212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:30:05.851409 14212 webserver.cc:460] Webserver started at http://127.0.98.1:36201/ using document root <none> and password file <none>
I20260925 16:30:05.852205 14212 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:05.852291 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:05.852557 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:05.855432 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "691388f799d8497a8b3f5500b22c0e94"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "f9fffa3b4a8d8d29d6fbc750aa91bc3e"
server_key_iv: "e2d1c524097053c220a811beced5fa1c"
server_key_version: "encryptionkey@0"
I20260925 16:30:05.856174 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "691388f799d8497a8b3f5500b22c0e94"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "f9fffa3b4a8d8d29d6fbc750aa91bc3e"
server_key_iv: "e2d1c524097053c220a811beced5fa1c"
server_key_version: "encryptionkey@0"
I20260925 16:30:05.861666 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:30:05.865646 14227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:05.867800 14212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:05.868037 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "691388f799d8497a8b3f5500b22c0e94"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "f9fffa3b4a8d8d29d6fbc750aa91bc3e"
server_key_iv: "e2d1c524097053c220a811beced5fa1c"
server_key_version: "encryptionkey@0"
I20260925 16:30:05.868217 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:05.897023 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:05.902001 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:05.902247 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:05.903095 14212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:05.904248 14212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:05.904302 14212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:05.904366 14212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:05.904395 14212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:05.915860 14212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39817
I20260925 16:30:05.915894 14340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39817 every 8 connection(s)
I20260925 16:30:05.917095 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:30:05.922238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14212
I20260925 16:30:05.922400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:30:05.922683 392 external_mini_cluster.cc:1428] Setting key d3d5d01160a7a703fcd1ed7a80bb9614
Sep 25 16:30:05 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:30:05.935810 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:05.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44183 (local address 127.0.98.62:45435)
0925 16:30:05.919956 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:05.919960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:05.921421 (+ 1461us) server_negotiation.cc:407] Connection header received
0925 16:30:05.922651 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:05.922656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:05.922734 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:05.922824 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:05.924800 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:05.925395 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:05.927396 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:05.927758 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:05.927883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:05.931384 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:05.931389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:05.931392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:05.931431 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:05.932913 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:05.933822 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:05.933825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:05.933828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:05.933932 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:05.934377 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:05.934381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:05.934382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:05.934614 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:05.934711 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:05.935490 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 16:30:05.935672 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:30:05.936911 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:05.920212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:44183)
0925 16:30:05.921168 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:05.921261 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:05.922364 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:05.923107 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:05.923118 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:05.923798 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:05.924579 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:05.924604 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:05.925541 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:05.925545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:05.927180 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:05.927210 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:05.927996 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:05.928000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:05.928224 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:05.928926 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:05.928947 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:05.931050 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:05.933085 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:05.933096 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:05.933122 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:05.933561 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:05.934053 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:05.934056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:05.934057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:05.934234 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:05.934724 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:05.934771 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:05.935141 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 16:30:05.935358 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 16:30:05.935779 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":177,"threads_started":1}
I20260925 16:30:05.938582 14341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45435
I20260925 16:30:05.939018 14341 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:05.939713 14341 heartbeater.cc:511] Master 127.0.98.62:45435 requested a full tablet report, sending...
I20260925 16:30:05.942238 14145 ts_manager.cc:194] Registered new tserver with Master: 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817)
I20260925 16:30:05.944308 14145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:30:05.956796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45435
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:06.102581 14346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:06.103004 14346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:06.103056 14346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:06.107882 14346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:06.108000 14346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:06.108071 14346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:30:06.114758 14346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:06.116326 14346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:06.117672 14346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:06.130574 14352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:06.130669 14346 server_base.cc:1034] running on GCE node
W20260925 16:30:06.130560 14351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:06.130565 14354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:06.131485 14346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:06.132295 14346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:06.133515 14346 hybrid_clock.cc:648] HybridClock initialized: now 1790353806133483 us; error 50 us; skew 500 ppm
Sep 25 16:30:06 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:06.136765 14346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:30:06.138074 14346 webserver.cc:460] Webserver started at http://127.0.98.2:34761/ using document root <none> and password file <none>
I20260925 16:30:06.138780 14346 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:06.138863 14346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:06.139098 14346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:06.141875 14346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "8f561c824da7c020f535c70432858b90"
server_key_iv: "b257017e7ad78f609919362d0e6a8d76"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.142836 14346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "8f561c824da7c020f535c70432858b90"
server_key_iv: "b257017e7ad78f609919362d0e6a8d76"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.149212 14346 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 16:30:06.154048 14361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.156307 14346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:06.156519 14346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "8f561c824da7c020f535c70432858b90"
server_key_iv: "b257017e7ad78f609919362d0e6a8d76"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.156677 14346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:06.171360 14346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:06.176494 14346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:06.176790 14346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:06.177696 14346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:06.179037 14346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:06.179103 14346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.179158 14346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:06.179174 14346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.191473 14346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38681
I20260925 16:30:06.191529 14474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38681 every 8 connection(s)
I20260925 16:30:06.192785 14346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:30:06.197190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14346
I20260925 16:30:06.197386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:30:06.197747 392 external_mini_cluster.cc:1428] Setting key a57c36a8678dea0adf1fed2e18afa1ba
Sep 25 16:30:06 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:30:06.215488 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.195790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47543 (local address 127.0.98.62:45435)
0925 16:30:06.196024 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.196029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.198190 (+ 2161us) server_negotiation.cc:407] Connection header received
0925 16:30:06.199656 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.199660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.199727 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.199842 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:06.201915 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.202578 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.204750 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.205078 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.205226 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.209865 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:06.209870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:06.209872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:06.209913 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:06.211906 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:06.212961 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:06.212965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:06.212967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:06.213096 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:06.213692 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:06.213700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:06.213702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:06.214128 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:06.214260 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.215118 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.215297 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 16:30:06.216698 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.196394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:47543)
0925 16:30:06.197816 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.198005 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.199196 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.200143 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.200156 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.200796 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:06.201596 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.201620 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.202792 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.202796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.204450 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.204476 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.205367 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.205372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.205679 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.206873 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:06.206907 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:06.209349 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:06.212129 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:06.212144 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:06.212169 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:06.212685 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:06.213243 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:06.213246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:06.213258 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:06.213558 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:06.214309 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:06.214317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:06.214764 (+ 447us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.215002 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.215439 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":395,"threads_started":1}
I20260925 16:30:06.218416 14475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45435
I20260925 16:30:06.218855 14475 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:06.219691 14475 heartbeater.cc:511] Master 127.0.98.62:45435 requested a full tablet report, sending...
I20260925 16:30:06.221490 14145 ts_manager.cc:194] Registered new tserver with Master: 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
I20260925 16:30:06.222536 14145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:30:06.233577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45435
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:06.367574 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:06.367894 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:06.367952 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:06.372339 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:06.372427 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:06.372493 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:30:06.378402 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:06.379598 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:06.380780 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:06.392524 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:06.392536 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:06.392655 14480 server_base.cc:1034] running on GCE node
W20260925 16:30:06.392536 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:06.393504 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:06.394551 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:06.395789 14480 hybrid_clock.cc:648] HybridClock initialized: now 1790353806395776 us; error 42 us; skew 500 ppm
Sep 25 16:30:06 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:06.399914 14480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:30:06.401273 14480 webserver.cc:460] Webserver started at http://127.0.98.3:33389/ using document root <none> and password file <none>
I20260925 16:30:06.402074 14480 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:06.402158 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:06.402423 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:06.405136 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "7d074605ae4f7bdbf55e955a66e39a03"
server_key_iv: "d798e3015fa2f278e0c8c3b10ad71f63"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.405890 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "7d074605ae4f7bdbf55e955a66e39a03"
server_key_iv: "d798e3015fa2f278e0c8c3b10ad71f63"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.411077 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:30:06.414753 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.416684 14480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:06.416924 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "7d074605ae4f7bdbf55e955a66e39a03"
server_key_iv: "d798e3015fa2f278e0c8c3b10ad71f63"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.417081 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:06.451915 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:06.456533 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:06.456789 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:06.457610 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:06.458797 14480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:06.458853 14480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.458917 14480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:06.458942 14480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.470597 14480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46677
I20260925 16:30:06.470862 14608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46677 every 8 connection(s)
I20260925 16:30:06.471822 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:30:06.472582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14480
I20260925 16:30:06.472718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:30:06.473220 392 external_mini_cluster.cc:1428] Setting key 572d6c2f846551f1df74bf704cc9b029
Sep 25 16:30:06 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:06.492491 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.474518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54511 (local address 127.0.98.62:45435)
0925 16:30:06.474685 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.474690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.476236 (+ 1546us) server_negotiation.cc:407] Connection header received
0925 16:30:06.477890 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.477894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.477975 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.478078 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:06.480066 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.480759 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.482895 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.483200 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.483315 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.487200 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:06.487205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:06.487208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:06.487278 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:06.489166 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:06.490043 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:06.490046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:06.490047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:06.490161 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:06.490708 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:06.490711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:06.490713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:06.491003 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:06.491109 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.492115 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.492303 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:30:06.493324 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.475208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:54511)
0925 16:30:06.475966 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.476083 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.477513 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.478438 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.478451 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.479172 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:06.479809 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.479830 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.481153 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.481157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.482654 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.482678 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.483415 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.483420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.483658 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.484545 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:06.484567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:06.486727 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:06.489394 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:06.489402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:06.489422 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:06.489918 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:06.490324 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:06.490328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:06.490330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:06.490541 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:06.491162 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:06.491169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:06.491517 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.491960 (+ 443us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.492327 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":161,"threads_started":1}
I20260925 16:30:06.495105 14609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45435
I20260925 16:30:06.495393 14609 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:06.495991 14609 heartbeater.cc:511] Master 127.0.98.62:45435 requested a full tablet report, sending...
I20260925 16:30:06.497583 14145 ts_manager.cc:194] Registered new tserver with Master: 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677)
I20260925 16:30:06.498556 14145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54511
I20260925 16:30:06.501664 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:30:06.515532 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.504989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.0.98.62:45435)
0925 16:30:06.505175 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.505181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.505650 (+ 469us) server_negotiation.cc:407] Connection header received
0925 16:30:06.505733 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.505737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.505896 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.505979 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:06.507249 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.507865 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.509029 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.509291 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.509420 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.511459 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:06.511464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:06.511467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:06.511520 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:06.513312 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:06.513971 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:06.513974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:06.513976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:06.514087 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:06.514390 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:06.514393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:06.514394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:06.514722 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:06.514853 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.515193 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.515356 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:30:06.520008 14145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:30:06.524405 14145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:30:06.543983 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.537604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46677 (local address 127.0.0.1:55384)
0925 16:30:06.537952 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.537964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.538042 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.538836 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.538839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.538858 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:06.538942 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.538948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.540376 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.540379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.541508 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.541515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.543068 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.543072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.543321 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.543402 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.543543 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.543769 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":123,"thread_start_us":119,"threads_started":1}
I20260925 16:30:06.544046 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.537023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.0.98.1:39817)
0925 16:30:06.537484 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.537490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.537810 (+ 320us) server_negotiation.cc:407] Connection header received
0925 16:30:06.538040 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.538046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.538218 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.538348 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:06.539360 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.540546 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.541748 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.542637 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.542763 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.542988 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.543744 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.543893 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":136,"threads_started":1}
I20260925 16:30:06.544090 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.536132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.0.1:40388)
0925 16:30:06.537148 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.537214 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.537390 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.538392 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.538395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.538414 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:06.538585 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.538594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.539961 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.539965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.541468 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.541483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.543068 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.543072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.543327 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.543386 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.543698 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.543869 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":83,"thread_start_us":150,"threads_started":1}
I20260925 16:30:06.544409 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.536196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39817 (local address 127.0.0.1:45440)
0925 16:30:06.537633 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.537646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.537875 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.538366 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.538373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.538396 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:06.538585 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.538594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.540667 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.540671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.541604 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.541618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.543068 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.543072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.543338 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.543400 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.543543 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.544275 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"mutex_wait_us":566,"thread_start_us":122,"threads_started":1}
I20260925 16:30:06.544863 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.537617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.0.98.3:46677)
0925 16:30:06.538166 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.538184 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.538217 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:30:06.538301 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.538305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.538476 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.538858 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:06.539046 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.540213 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.541648 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.542637 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.542769 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.542988 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.544538 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.544689 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":215,"threads_started":1}
I20260925 16:30:06.544981 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.536369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.0.98.2:38681)
0925 16:30:06.537107 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.537121 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.537324 (+ 203us) server_negotiation.cc:407] Connection header received
0925 16:30:06.537647 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.537655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.538014 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.538185 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:06.538710 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.539827 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.541660 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.542640 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.542763 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.542988 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.544593 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.544791 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":184,"threads_started":1}
I20260925 16:30:06.548135 14275 tablet_service.cc:1467] Processing CreateTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 (DEFAULT_TABLE table=test-table [id=35ff320609614f8fbdaf81e76eb53c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:06.548791 14409 tablet_service.cc:1467] Processing CreateTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 (DEFAULT_TABLE table=test-table [id=35ff320609614f8fbdaf81e76eb53c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:06.550511 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7394169c0d44dbf96e3504d37bfe9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:06.550499 14543 tablet_service.cc:1467] Processing CreateTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 (DEFAULT_TABLE table=test-table [id=35ff320609614f8fbdaf81e76eb53c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:06.551556 14275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7394169c0d44dbf96e3504d37bfe9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:06.551730 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7394169c0d44dbf96e3504d37bfe9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:06.562752 14626 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Bootstrap starting.
I20260925 16:30:06.563282 14628 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap starting.
I20260925 16:30:06.564270 14627 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Bootstrap starting.
I20260925 16:30:06.565508 14626 tablet_bootstrap.cc:654] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:06.566478 14628 tablet_bootstrap.cc:654] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:06.566946 14626 log.cc:824] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:06.567219 14627 tablet_bootstrap.cc:654] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:06.567433 14628 log.cc:824] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:06.568107 14627 log.cc:824] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:06.569865 14626 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: No bootstrap required, opened a new log
I20260925 16:30:06.570122 14628 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: No bootstrap required, opened a new log
I20260925 16:30:06.570122 14626 ts_tablet_manager.cc:1397] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 16:30:06.570251 14627 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: No bootstrap required, opened a new log
I20260925 16:30:06.570355 14628 ts_tablet_manager.cc:1397] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:30:06.570638 14627 ts_tablet_manager.cc:1397] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:30:06.574863 14628 raft_consensus.cc:348] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.575206 14628 raft_consensus.cc:374] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:06.575268 14628 raft_consensus.cc:729] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691388f799d8497a8b3f5500b22c0e94, State: Initialized, Role: FOLLOWER
I20260925 16:30:06.575057 14626 raft_consensus.cc:348] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.575057 14627 raft_consensus.cc:348] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.575438 14627 raft_consensus.cc:374] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:06.575438 14626 raft_consensus.cc:374] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:06.575511 14626 raft_consensus.cc:729] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0082646c9c7a4237a5bc6f1cb1fc6c77, State: Initialized, Role: FOLLOWER
I20260925 16:30:06.575518 14627 raft_consensus.cc:729] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ad0a4c9d14e489d08e36ba1189ba9, State: Initialized, Role: FOLLOWER
I20260925 16:30:06.575915 14628 consensus_queue.cc:260] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.576092 14627 consensus_queue.cc:260] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.576092 14626 consensus_queue.cc:260] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.576911 14341 heartbeater.cc:503] Master 127.0.98.62:45435 was elected leader, sending a full tablet report...
I20260925 16:30:06.577279 14628 ts_tablet_manager.cc:1428] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:30:06.577643 14627 ts_tablet_manager.cc:1428] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:30:06.578195 14475 heartbeater.cc:503] Master 127.0.98.62:45435 was elected leader, sending a full tablet report...
I20260925 16:30:06.578835 14626 ts_tablet_manager.cc:1428] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260925 16:30:06.579228 14609 heartbeater.cc:503] Master 127.0.98.62:45435 was elected leader, sending a full tablet report...
I20260925 16:30:06.656070 14632 raft_consensus.cc:482] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:06.656368 14632 raft_consensus.cc:504] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.657922 14632 leader_election.cc:288] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681), 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677)
I20260925 16:30:06.662442 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.658302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46677 (local address 127.0.98.1:59983)
0925 16:30:06.658773 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.658787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.658930 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.659234 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.659238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.659253 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:06.659418 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.659425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.660208 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.660211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.661182 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.661193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.661906 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.661909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.662015 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.662046 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.662123 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.662229 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":159,"threads_started":1}
I20260925 16:30:06.662420 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.658439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59983 (local address 127.0.98.3:46677)
0925 16:30:06.658595 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.658600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.658893 (+ 293us) server_negotiation.cc:407] Connection header received
0925 16:30:06.659071 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.659073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.659132 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.659211 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:06.659524 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.660092 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.661306 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.661779 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.661887 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.661991 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.662175 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.662273 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:30:06.662943 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.658432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36065 (local address 127.0.98.2:38681)
0925 16:30:06.658595 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.658600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.658988 (+ 388us) server_negotiation.cc:407] Connection header received
0925 16:30:06.659130 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.659134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.659191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.659365 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:06.659860 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.660392 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.661618 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.662263 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.662372 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.662450 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.662692 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.662787 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:30:06.663434 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.658302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.98.1:36065)
0925 16:30:06.658882 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.658897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:06.659020 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:06.659451 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:06.659456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:06.659491 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:06.659686 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.659694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.660530 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.660534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.661471 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:06.661483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.662403 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.662407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:06.662522 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.662554 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 16:30:06.663176 (+ 622us) client_negotiation.cc:240] Negotiation successful
0925 16:30:06.663274 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":202,"thread_start_us":140,"threads_started":1}
I20260925 16:30:06.663410 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7394169c0d44dbf96e3504d37bfe9d7" candidate_uuid: "691388f799d8497a8b3f5500b22c0e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" is_pre_election: true
I20260925 16:30:06.663862 14563 raft_consensus.cc:2454] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 691388f799d8497a8b3f5500b22c0e94 in term 0.
I20260925 16:30:06.664254 14429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7394169c0d44dbf96e3504d37bfe9d7" candidate_uuid: "691388f799d8497a8b3f5500b22c0e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" is_pre_election: true
I20260925 16:30:06.664454 14231 leader_election.cc:302] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 691388f799d8497a8b3f5500b22c0e94; no voters:
I20260925 16:30:06.664618 14429 raft_consensus.cc:2454] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 691388f799d8497a8b3f5500b22c0e94 in term 0.
I20260925 16:30:06.664852 14632 raft_consensus.cc:2781] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:30:06.664932 14632 raft_consensus.cc:482] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:30:06.664983 14632 raft_consensus.cc:3037] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:06.666790 14632 raft_consensus.cc:504] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.667323 14632 leader_election.cc:288] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 1 election: Requested vote from peers 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681), 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677)
I20260925 16:30:06.667937 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7394169c0d44dbf96e3504d37bfe9d7" candidate_uuid: "691388f799d8497a8b3f5500b22c0e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77"
I20260925 16:30:06.667965 14429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7394169c0d44dbf96e3504d37bfe9d7" candidate_uuid: "691388f799d8497a8b3f5500b22c0e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
I20260925 16:30:06.668144 14563 raft_consensus.cc:3037] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:06.668144 14429 raft_consensus.cc:3037] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Advancing to term 1
W20260925 16:30:06.669032 14342 tablet.cc:2369] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:06.669778 14429 raft_consensus.cc:2454] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691388f799d8497a8b3f5500b22c0e94 in term 1.
I20260925 16:30:06.670083 14563 raft_consensus.cc:2454] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691388f799d8497a8b3f5500b22c0e94 in term 1.
I20260925 16:30:06.670342 14230 leader_election.cc:302] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 691388f799d8497a8b3f5500b22c0e94, 8e5ad0a4c9d14e489d08e36ba1189ba9; no voters:
I20260925 16:30:06.670619 14632 raft_consensus.cc:2781] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:06.670925 14632 raft_consensus.cc:686] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 LEADER]: Becoming Leader. State: Replica: 691388f799d8497a8b3f5500b22c0e94, State: Running, Role: LEADER
I20260925 16:30:06.671386 14632 consensus_queue.cc:237] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:06.676915 14149 catalog_manager.cc:5495] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "691388f799d8497a8b3f5500b22c0e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } health_report { overall_health: UNKNOWN } } }
W20260925 16:30:06.694505 14476 tablet.cc:2369] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:06.710273 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.704932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.0.98.1:39817)
0925 16:30:06.705206 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:06.705211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:06.705252 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:30:06.705391 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:06.705394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:06.705476 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:06.705590 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:30:06.706225 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.706879 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.707986 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:06.708266 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:06.708427 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:06.709100 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:30:06.709678 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:30:06.709884 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:06.709993 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 16:30:06.710091 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 16:30:06.718876 14563 raft_consensus.cc:1261] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Refusing update from remote peer 691388f799d8497a8b3f5500b22c0e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:06.719110 14429 raft_consensus.cc:1261] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Refusing update from remote peer 691388f799d8497a8b3f5500b22c0e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:06.720095 14637 consensus_queue.cc:1035] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:06.720434 14632 consensus_queue.cc:1035] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:30:06.723064 14610 tablet.cc:2369] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:06.729604 14640 mvcc.cc:205] Tried to move back new op lower bound from 7333289192310849536 to 7333289192130596864. Current Snapshot: MvccSnapshot[applied={T|T < 7333289192310849536}]
I20260925 16:30:06.731446 14641 mvcc.cc:205] Tried to move back new op lower bound from 7333289192310849536 to 7333289192130596864. Current Snapshot: MvccSnapshot[applied={T|T < 7333289192310849536}]
I20260925 16:30:06.743662 14149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:06.744005 14149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:06.747643 14149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Sending DeleteTablet for 3 replicas of tablet e7394169c0d44dbf96e3504d37bfe9d7
I20260925 16:30:06.749080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14212
I20260925 16:30:06.749105 14275 tablet_service.cc:1514] Processing DeleteTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45440
I20260925 16:30:06.749096 14409 tablet_service.cc:1514] Processing DeleteTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40388
I20260925 16:30:06.749840 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55384
I20260925 16:30:06.750519 14651 tablet_replica.cc:330] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: stopping tablet replica
I20260925 16:30:06.751005 14651 raft_consensus.cc:2229] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:06.751446 14650 tablet_replica.cc:330] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: stopping tablet replica
I20260925 16:30:06.751500 14651 raft_consensus.cc:2258] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:06.751873 14650 raft_consensus.cc:2229] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:06.752261 14650 raft_consensus.cc:2258] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:06.753818 14650 ts_tablet_manager.cc:1905] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:06.753873 14651 ts_tablet_manager.cc:1905] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:06.756413 14650 ts_tablet_manager.cc:1918] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:06.756546 14650 log.cc:1196] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/e7394169c0d44dbf96e3504d37bfe9d7
I20260925 16:30:06.756824 14650 ts_tablet_manager.cc:1939] T e7394169c0d44dbf96e3504d37bfe9d7 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Deleting consensus metadata
I20260925 16:30:06.757643 14651 ts_tablet_manager.cc:1918] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:06.757841 14651 log.cc:1196] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/e7394169c0d44dbf96e3504d37bfe9d7
I20260925 16:30:06.758201 14651 ts_tablet_manager.cc:1939] T e7394169c0d44dbf96e3504d37bfe9d7 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Deleting consensus metadata
I20260925 16:30:06.758952 14136 catalog_manager.cc:4841] TS 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681): tablet e7394169c0d44dbf96e3504d37bfe9d7 (table test-table [id=35ff320609614f8fbdaf81e76eb53c90]) successfully deleted
W20260925 16:30:06.759330 14137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39817 (error 108)
I20260925 16:30:06.759697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14346
I20260925 16:30:06.760083 14137 catalog_manager.cc:4841] TS 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): tablet e7394169c0d44dbf96e3504d37bfe9d7 (table test-table [id=35ff320609614f8fbdaf81e76eb53c90]) successfully deleted
I20260925 16:30:06.761466 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:06.760968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39817 (local address 127.0.0.1:45468)
0925 16:30:06.761197 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:06.761329 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 16:30:06.761739 14137 catalog_manager.cc:4584] TS 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7394169c0d44dbf96e3504d37bfe9d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39817: connect: Connection refused (error 111)
I20260925 16:30:06.767200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14480
I20260925 16:30:06.774013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14118
I20260925 16:30:06.781956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--webserver_interface=127.0.98.62
--webserver_port=32791
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:06.907262 14653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:06.907591 14653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:06.907635 14653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:06.911990 14653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:06.912082 14653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:06.912101 14653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:06.912117 14653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:06.912132 14653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:06.917917 14653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:06.919137 14653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:06.920240 14653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:30:06.936371 14653 server_base.cc:1034] running on GCE node
W20260925 16:30:06.936285 14662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:06.936316 14660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:06.936290 14659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:06.937577 14653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:06.939023 14653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:06.940392 14653 hybrid_clock.cc:648] HybridClock initialized: now 1790353806940365 us; error 59 us; skew 500 ppm
Sep 25 16:30:06 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:06.943694 14653 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:06.945278 14653 webserver.cc:460] Webserver started at http://127.0.98.62:32791/ using document root <none> and password file <none>
I20260925 16:30:06.946066 14653 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:06.946164 14653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:06.951581 14653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:06.955399 14669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:06.957404 14653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260925 16:30:06.957654 14653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "51fc0e88189044a0b69a2ab7ae766cac"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "19371018548ef46a98dcd5ae96602859"
server_key_iv: "22564d5419d62d02be2d8534bd3ef751"
server_key_version: "encryptionkey@0"
I20260925 16:30:06.958503 14653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:06.974922 14653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:06.978667 14653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:06.978865 14653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:06.987397 14653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45435
I20260925 16:30:06.987859 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45435 every 8 connection(s)
I20260925 16:30:06.988768 14653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:06.991338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14653
I20260925 16:30:06.991854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39817
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45435
--webserver_port=36201
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:06.995690 14722 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:30:06.998428 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap starting.
I20260925 16:30:07.022136 14722 log.cc:824] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:07.033720 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:30:07.034286 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap complete.
I20260925 16:30:07.039567 14722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:07.040190 14722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fc0e88189044a0b69a2ab7ae766cac, State: Initialized, Role: FOLLOWER
I20260925 16:30:07.040884 14722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:07.041044 14722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:30:07.041123 14722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:30:07.041232 14722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:30:07.044162 14722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:07.044729 14722 leader_election.cc:302] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fc0e88189044a0b69a2ab7ae766cac; no voters:
I20260925 16:30:07.045235 14722 leader_election.cc:288] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:30:07.045389 14726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:30:07.045635 14726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 LEADER]: Becoming Leader. State: Replica: 51fc0e88189044a0b69a2ab7ae766cac, State: Running, Role: LEADER
I20260925 16:30:07.046090 14726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:07.047060 14722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:07.048445 14728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fc0e88189044a0b69a2ab7ae766cac. Latest consensus state: current_term: 2 leader_uuid: "51fc0e88189044a0b69a2ab7ae766cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260925 16:30:07.048660 14728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:07.048434 14727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "51fc0e88189044a0b69a2ab7ae766cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260925 16:30:07.048750 14727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:07.049189 14735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:07.055486 14735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:07.056370 14735 catalog_manager.cc:1259] Loaded cluster ID: ff3f731bc8ba42ceaa1f349f8afd1096
I20260925 16:30:07.056454 14735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:30:07.058200 14735 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 16:30:07.061131 14735 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 16:30:07.068994 14735 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:30:07.070271 14735 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Loaded TSK: 0
I20260925 16:30:07.071826 14735 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:30:07.151115 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:07.151502 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:07.151574 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:07.156366 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:07.156477 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:07.156579 14724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:30:07.163242 14724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36201
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:07.164772 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:07.166168 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:07.178296 14752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:07.178293 14749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:07.178293 14750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:07.179083 14724 server_base.cc:1034] running on GCE node
I20260925 16:30:07.179582 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:07.180495 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:07.181684 14724 hybrid_clock.cc:648] HybridClock initialized: now 1790353807181669 us; error 38 us; skew 500 ppm
Sep 25 16:30:07 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:07.185526 14724 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:30:07.187191 14724 webserver.cc:460] Webserver started at http://127.0.98.1:36201/ using document root <none> and password file <none>
I20260925 16:30:07.188115 14724 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:07.188265 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:07.194753 14724 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:07.198153 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.200275 14724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:07.200484 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "691388f799d8497a8b3f5500b22c0e94"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "f9fffa3b4a8d8d29d6fbc750aa91bc3e"
server_key_iv: "e2d1c524097053c220a811beced5fa1c"
server_key_version: "encryptionkey@0"
I20260925 16:30:07.201304 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:07.220691 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:07.225664 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:07.226166 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:07.227301 14724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:07.228845 14766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:30:07.234614 14724 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:30:07.234763 14724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 16:30:07.234838 14724 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:30:07.237011 14724 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:30:07.237088 14724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 16:30:07.237614 14766 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap starting.
I20260925 16:30:07.248896 14724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39817
I20260925 16:30:07.249028 14873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39817 every 8 connection(s)
I20260925 16:30:07.250298 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:30:07.251748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14724
I20260925 16:30:07.252187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38681
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45435
--webserver_port=34761
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:30:07 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:07.266192 14766 log.cc:824] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:07.272536 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.253137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56137 (local address 127.0.98.62:45435)
0925 16:30:07.253676 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.253701 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.254556 (+ 855us) server_negotiation.cc:407] Connection header received
0925 16:30:07.256138 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.256148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.256484 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.256811 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:07.258094 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.259095 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.260393 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.260865 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.261033 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.265172 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:07.265184 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:07.265208 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:07.265279 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:07.267789 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.268725 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.268733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.268740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.268972 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.269462 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.269466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.269468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.270059 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:07.270221 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.271118 (+ 897us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.271532 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":135,"threads_started":1}
I20260925 16:30:07.273048 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.253396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:56137)
0925 16:30:07.254213 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.254295 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.255639 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.256907 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.256916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.257371 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:07.257907 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.257927 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.259269 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.259272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.260258 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.260271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.261044 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.261049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.261240 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.262051 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:07.262086 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:07.264751 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:07.268022 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:07.268032 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:07.268055 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:07.268511 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:07.269097 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:07.269101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:07.269103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:07.269319 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:07.270222 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:07.270231 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:07.270660 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.270900 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.271685 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":240,"threads_started":1}
I20260925 16:30:07.276665 14874 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45435
I20260925 16:30:07.276786 14766 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:30:07.276994 14874 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:07.277201 14766 tablet_bootstrap.cc:492] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap complete.
I20260925 16:30:07.277650 14874 heartbeater.cc:511] Master 127.0.98.62:45435 requested a full tablet report, sending...
I20260925 16:30:07.278561 14766 ts_tablet_manager.cc:1397] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Time spent bootstrapping tablet: real 0.041s user 0.020s sys 0.016s
I20260925 16:30:07.281100 14686 ts_manager.cc:194] Registered new tserver with Master: 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817)
I20260925 16:30:07.281714 14766 raft_consensus.cc:348] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:07.282377 14766 raft_consensus.cc:729] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691388f799d8497a8b3f5500b22c0e94, State: Initialized, Role: FOLLOWER
I20260925 16:30:07.282954 14766 consensus_queue.cc:260] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } } peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } }
I20260925 16:30:07.284322 14766 ts_tablet_manager.cc:1428] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 16:30:07.285796 14686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56137
I20260925 16:30:07.289103 14874 heartbeater.cc:503] Master 127.0.98.62:45435 was elected leader, sending a full tablet report...
I20260925 16:30:07.295145 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.285169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39817 (local address 127.0.0.1:45480)
0925 16:30:07.285735 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.285866 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.286146 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.290283 (+ 4137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.290291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.290330 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:07.290555 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.290565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.292092 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.292097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.293140 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.293161 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.294490 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.294493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.294644 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.294692 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.294835 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.294972 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
I20260925 16:30:07.295246 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.285300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.98.1:39817)
0925 16:30:07.289565 (+ 4265us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.289580 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.289614 (+ 34us) server_negotiation.cc:407] Connection header received
0925 16:30:07.289716 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.289723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.290043 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.290248 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:07.290694 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.291836 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.293352 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.294271 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.294413 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.294617 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.294896 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.295036 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4120,"thread_start_us":145,"threads_started":1}
I20260925 16:30:07.296960 14808 tablet_service.cc:1514] Processing DeleteTablet for tablet e7394169c0d44dbf96e3504d37bfe9d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45480
I20260925 16:30:07.298173 14883 tablet_replica.cc:330] stopping tablet replica
I20260925 16:30:07.298664 14883 raft_consensus.cc:2229] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:07.299023 14883 raft_consensus.cc:2258] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:07.300590 14883 ts_tablet_manager.cc:1905] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:07.307194 14883 ts_tablet_manager.cc:1918] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:07.307376 14883 log.cc:1196] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/e7394169c0d44dbf96e3504d37bfe9d7
I20260925 16:30:07.307668 14883 ts_tablet_manager.cc:1939] T e7394169c0d44dbf96e3504d37bfe9d7 P 691388f799d8497a8b3f5500b22c0e94: Deleting consensus metadata
I20260925 16:30:07.310734 14673 catalog_manager.cc:4841] TS 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817): tablet e7394169c0d44dbf96e3504d37bfe9d7 (table test-table [id=35ff320609614f8fbdaf81e76eb53c90]) successfully deleted
W20260925 16:30:07.408464 14877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:07.408874 14877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:07.408932 14877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:07.413659 14877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:07.413789 14877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:07.413874 14877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:30:07.420401 14877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34761
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:07.422222 14877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:07.423790 14877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:07.437623 14888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:07.437638 14889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:07.437623 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:07.438133 14877 server_base.cc:1034] running on GCE node
I20260925 16:30:07.438714 14877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:07.439531 14877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:07.440739 14877 hybrid_clock.cc:648] HybridClock initialized: now 1790353807440705 us; error 45 us; skew 500 ppm
Sep 25 16:30:07 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:07.444111 14877 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:30:07.445530 14877 webserver.cc:460] Webserver started at http://127.0.98.2:34761/ using document root <none> and password file <none>
I20260925 16:30:07.446496 14877 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:07.446615 14877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:07.451984 14877 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 16:30:07.455075 14898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.457093 14877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:07.457276 14877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "8f561c824da7c020f535c70432858b90"
server_key_iv: "b257017e7ad78f609919362d0e6a8d76"
server_key_version: "encryptionkey@0"
I20260925 16:30:07.458081 14877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:07.476507 14877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:07.480597 14877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:07.480805 14877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:07.481567 14877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:07.482730 14877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:07.482784 14877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.482831 14877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:07.482853 14877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.494928 14877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38681
I20260925 16:30:07.495034 15011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38681 every 8 connection(s)
I20260925 16:30:07.496411 14877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:30:07.505069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 14877
I20260925 16:30:07.505618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46677
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45435
--webserver_port=33389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:30:07 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:07.516343 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.499117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33947 (local address 127.0.98.62:45435)
0925 16:30:07.499412 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.499419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.501293 (+ 1874us) server_negotiation.cc:407] Connection header received
0925 16:30:07.502554 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.502559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.502633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.502710 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:07.504243 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.504836 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.506326 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.506581 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.506680 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.511041 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:07.511048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:07.511053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:07.511114 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:07.512844 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.513893 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.513898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.513899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.514021 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.514608 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.514611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.514613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.514918 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:07.515042 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.515935 (+ 893us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.516166 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 16:30:07.517544 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.499772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:33947)
0925 16:30:07.501057 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.501128 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.502225 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.502955 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.502964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.503411 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:07.504051 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.504068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.504977 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.504981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.506178 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.506194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.506761 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.506765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.507009 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.507983 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:07.508018 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:07.510644 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:07.513070 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:07.513080 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:07.513106 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:07.513626 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:07.514147 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:07.514151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:07.514154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:07.514414 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:07.515068 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:07.515076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:07.515540 (+ 464us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.515815 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.516293 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":661,"threads_started":1}
I20260925 16:30:07.519395 15012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45435
I20260925 16:30:07.519759 15012 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:07.520480 15012 heartbeater.cc:511] Master 127.0.98.62:45435 requested a full tablet report, sending...
I20260925 16:30:07.522116 14686 ts_manager.cc:194] Registered new tserver with Master: 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
I20260925 16:30:07.522975 14686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33947
W20260925 16:30:07.652769 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:07.653229 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:07.653285 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:07.658344 15015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:07.658461 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:07.658536 15015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:30:07.665469 15015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33389
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:07.667137 15015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:07.668582 15015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:07.681352 15023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:07.681375 15021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:07.681375 15020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:07.681857 15015 server_base.cc:1034] running on GCE node
I20260925 16:30:07.682528 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:07.683483 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:07.684770 15015 hybrid_clock.cc:648] HybridClock initialized: now 1790353807684734 us; error 56 us; skew 500 ppm
Sep 25 16:30:07 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:07.688593 15015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:30:07.690026 15015 webserver.cc:460] Webserver started at http://127.0.98.3:33389/ using document root <none> and password file <none>
I20260925 16:30:07.690824 15015 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:07.690932 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:07.696614 15015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:30:07.700330 15030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.702533 15015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:07.702739 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "7d074605ae4f7bdbf55e955a66e39a03"
server_key_iv: "d798e3015fa2f278e0c8c3b10ad71f63"
server_key_version: "encryptionkey@0"
I20260925 16:30:07.703609 15015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:07.736557 15015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:07.741055 15015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:07.741307 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:07.742164 15015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:07.743269 15015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:07.743322 15015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.743384 15015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:07.743408 15015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:07.755585 15015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46677
I20260925 16:30:07.755620 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46677 every 8 connection(s)
I20260925 16:30:07.756963 15015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:30:07.767096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15015
Sep 25 16:30:07 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:07.775527 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.759434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44839 (local address 127.0.98.62:45435)
0925 16:30:07.759749 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.759753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.761151 (+ 1398us) server_negotiation.cc:407] Connection header received
0925 16:30:07.762377 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.762381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.762459 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.762574 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:07.764149 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.764787 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.766609 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.766866 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.766972 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.770523 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:07.770528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:07.770531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:07.770565 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:07.772382 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.772964 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.772968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.772971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.773082 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.773741 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.773744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.773746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.774092 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:07.774202 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.775156 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.775311 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 16:30:07.776779 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.759918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:44839)
0925 16:30:07.760876 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.760938 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.762078 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.762835 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.762843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.763283 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:07.763962 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.763981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.764983 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.764986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.766435 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.766457 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.767146 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.767153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.767451 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.768305 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:07.768329 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:07.770267 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:07.772523 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:07.772529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:07.772546 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:07.772839 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:07.773238 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:07.773242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:07.773244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:07.773581 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:07.774255 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:07.774266 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:07.774786 (+ 520us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.775047 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.775511 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":157,"threads_started":1}
I20260925 16:30:07.777959 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.768219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.98.62:45435)
0925 16:30:07.768538 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.768542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.768557 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:30:07.768694 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.768697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.768755 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.768839 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:07.769890 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.770606 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.771589 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.771864 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.771954 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.773928 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:07.773932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:07.773935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:07.773956 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:07.775901 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.776350 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.776354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.776356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.776435 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:07.776698 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:07.776702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:07.776704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:07.776929 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:07.777158 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.777562 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.777783 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260925 16:30:07.778419 15144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45435
I20260925 16:30:07.778721 15144 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:07.779340 15144 heartbeater.cc:511] Master 127.0.98.62:45435 requested a full tablet report, sending...
I20260925 16:30:07.780862 14686 ts_manager.cc:194] Registered new tserver with Master: 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677)
I20260925 16:30:07.781810 14686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44839
I20260925 16:30:07.782171 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:30:07.788092 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.783058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.0.98.62:45435)
0925 16:30:07.783208 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.783213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.783481 (+ 268us) server_negotiation.cc:407] Connection header received
0925 16:30:07.783647 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.783651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.783826 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.784006 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:30:07.784329 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.785132 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.786244 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.786514 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.786619 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.787036 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:30:07.787537 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:30:07.787656 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.787811 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.787915 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:30:07.789105 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:30:07.792898 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:30:07.803905 14808 tablet_service.cc:1467] Processing CreateTablet for tablet cadce69f81e3485f9727ab62a8eb21d5 (DEFAULT_TABLE table=test-table [id=f2100601e4b54acdacfea3f08b5c88a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:07.804668 14808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadce69f81e3485f9727ab62a8eb21d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:07.808763 15157 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap starting.
I20260925 16:30:07.809813 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.803013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.98.2:38681)
0925 16:30:07.803571 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.803585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.803696 (+ 111us) server_negotiation.cc:407] Connection header received
0925 16:30:07.803846 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.803853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.804133 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.804329 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:07.804869 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.806087 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.807731 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.808865 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.808980 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.809183 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.809490 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.809625 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":160,"threads_started":1}
I20260925 16:30:07.809808 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.802498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46677 (local address 127.0.0.1:55398)
0925 16:30:07.803155 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.803173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.803285 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.804215 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.804218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.804238 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:07.804392 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.804398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.806316 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.806320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.807693 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.807710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.809169 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.809173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.809285 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.809323 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.809400 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.809519 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":27,"thread_start_us":247,"threads_started":1}
I20260925 16:30:07.809937 15157 tablet_bootstrap.cc:654] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:07.810216 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.802759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.0.1:40398)
0925 16:30:07.803580 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.803593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.803706 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.804571 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.804576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.804603 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:07.804733 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.804740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.806246 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.806297 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.807347 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.807361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.809244 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.809247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.809332 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.809354 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.809422 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.809542 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":34,"thread_start_us":127,"threads_started":1}
I20260925 16:30:07.810343 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.802846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.0.98.3:46677)
0925 16:30:07.803577 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.803592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.803627 (+ 35us) server_negotiation.cc:407] Connection header received
0925 16:30:07.803748 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.803756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.804016 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.805096 (+ 1080us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:07.805298 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.806173 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.807927 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.809005 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.809724 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.809950 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.810028 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.810159 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":183,"threads_started":1}
I20260925 16:30:07.811470 15157 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: No bootstrap required, opened a new log
I20260925 16:30:07.811599 15157 ts_tablet_manager.cc:1397] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 16:30:07.812058 15157 raft_consensus.cc:348] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.812201 15157 raft_consensus.cc:374] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:07.812239 15157 raft_consensus.cc:729] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691388f799d8497a8b3f5500b22c0e94, State: Initialized, Role: FOLLOWER
I20260925 16:30:07.812371 15157 consensus_queue.cc:260] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.812745 15157 ts_tablet_manager.cc:1428] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 16:30:07.814041 15078 tablet_service.cc:1467] Processing CreateTablet for tablet cadce69f81e3485f9727ab62a8eb21d5 (DEFAULT_TABLE table=test-table [id=f2100601e4b54acdacfea3f08b5c88a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:07.814199 14946 tablet_service.cc:1467] Processing CreateTablet for tablet cadce69f81e3485f9727ab62a8eb21d5 (DEFAULT_TABLE table=test-table [id=f2100601e4b54acdacfea3f08b5c88a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:07.815199 15078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadce69f81e3485f9727ab62a8eb21d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:07.815613 14946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadce69f81e3485f9727ab62a8eb21d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:07.824146 15160 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Bootstrap starting.
I20260925 16:30:07.824203 15161 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Bootstrap starting.
I20260925 16:30:07.826861 15160 tablet_bootstrap.cc:654] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:07.827405 15161 tablet_bootstrap.cc:654] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:07.828150 15160 log.cc:824] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:07.828341 15161 log.cc:824] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:07.830866 15160 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: No bootstrap required, opened a new log
I20260925 16:30:07.831130 15160 ts_tablet_manager.cc:1397] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:30:07.831226 15161 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: No bootstrap required, opened a new log
I20260925 16:30:07.831488 15161 ts_tablet_manager.cc:1397] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:30:07.834862 15161 raft_consensus.cc:348] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.834874 15160 raft_consensus.cc:348] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.835175 15160 raft_consensus.cc:374] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:07.835175 15161 raft_consensus.cc:374] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:07.835213 15160 raft_consensus.cc:729] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0082646c9c7a4237a5bc6f1cb1fc6c77, State: Initialized, Role: FOLLOWER
I20260925 16:30:07.835238 15161 raft_consensus.cc:729] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5ad0a4c9d14e489d08e36ba1189ba9, State: Initialized, Role: FOLLOWER
I20260925 16:30:07.835768 15161 consensus_queue.cc:260] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.835863 15160 consensus_queue.cc:260] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.836568 15012 heartbeater.cc:503] Master 127.0.98.62:45435 was elected leader, sending a full tablet report...
I20260925 16:30:07.837087 15161 ts_tablet_manager.cc:1428] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:30:07.837131 15160 ts_tablet_manager.cc:1428] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 16:30:07.837116 15164 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:07.837442 15164 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.837738 15144 heartbeater.cc:503] Master 127.0.98.62:45435 was elected leader, sending a full tablet report...
I20260925 16:30:07.839267 15164 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681), 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817)
I20260925 16:30:07.844041 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.839525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.98.3:33659)
0925 16:30:07.840133 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.840152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.840266 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.840685 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.840689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.840709 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:07.840842 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.840850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.841745 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.841748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.842880 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.842897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.843609 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.843613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.843729 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.843753 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.843819 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.843900 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516}
I20260925 16:30:07.844066 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.839814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33659 (local address 127.0.98.2:38681)
0925 16:30:07.840156 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.840161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.840349 (+ 188us) server_negotiation.cc:407] Connection header received
0925 16:30:07.840410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.840413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.840467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.840536 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:07.840969 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.841629 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.843034 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.843486 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.843576 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.843666 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.843841 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.843927 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260925 16:30:07.844825 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.840394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54183 (local address 127.0.98.1:39817)
0925 16:30:07.840682 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.840698 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.841000 (+ 302us) server_negotiation.cc:407] Connection header received
0925 16:30:07.841060 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.841063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.841114 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.841219 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:07.841827 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.842354 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.843679 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.844266 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.844376 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.844451 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.844619 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.844698 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260925 16:30:07.844848 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.839775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39817 (local address 127.0.98.3:54183)
0925 16:30:07.840822 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.840835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:07.840927 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:07.841275 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:07.841279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:07.841304 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:07.841669 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.841678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.842468 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.842471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.843539 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:07.843550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.844397 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.844401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:07.844497 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.844522 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:30:07.844594 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:30:07.844680 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":147,"threads_started":1}
I20260925 16:30:07.845047 14966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadce69f81e3485f9727ab62a8eb21d5" candidate_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" is_pre_election: true
I20260925 16:30:07.845404 14828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadce69f81e3485f9727ab62a8eb21d5" candidate_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691388f799d8497a8b3f5500b22c0e94" is_pre_election: true
I20260925 16:30:07.845520 14966 raft_consensus.cc:2454] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0082646c9c7a4237a5bc6f1cb1fc6c77 in term 0.
I20260925 16:30:07.845728 14828 raft_consensus.cc:2454] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0082646c9c7a4237a5bc6f1cb1fc6c77 in term 0.
I20260925 16:30:07.846184 15033 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 8e5ad0a4c9d14e489d08e36ba1189ba9; no voters:
I20260925 16:30:07.846551 15164 raft_consensus.cc:2781] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:30:07.846642 15164 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:30:07.846684 15164 raft_consensus.cc:3037] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:07.848078 15164 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.848932 15164 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [CANDIDATE]: Term 1 election: Requested vote from peers 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681), 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817)
I20260925 16:30:07.849463 14828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadce69f81e3485f9727ab62a8eb21d5" candidate_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691388f799d8497a8b3f5500b22c0e94"
I20260925 16:30:07.849529 14966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadce69f81e3485f9727ab62a8eb21d5" candidate_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
I20260925 16:30:07.849627 14828 raft_consensus.cc:3037] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:07.849712 14966 raft_consensus.cc:3037] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:07.851097 14828 raft_consensus.cc:2454] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0082646c9c7a4237a5bc6f1cb1fc6c77 in term 1.
I20260925 16:30:07.851130 14966 raft_consensus.cc:2454] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0082646c9c7a4237a5bc6f1cb1fc6c77 in term 1.
I20260925 16:30:07.851593 15034 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 691388f799d8497a8b3f5500b22c0e94; no voters:
I20260925 16:30:07.851926 15164 raft_consensus.cc:2781] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:07.852072 15164 raft_consensus.cc:686] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 LEADER]: Becoming Leader. State: Replica: 0082646c9c7a4237a5bc6f1cb1fc6c77, State: Running, Role: LEADER
I20260925 16:30:07.852450 15164 consensus_queue.cc:237] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:07.856739 14682 catalog_manager.cc:5495] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20260925 16:30:07.881542 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.877114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.0.98.3:46677)
0925 16:30:07.877294 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:07.877300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:07.877322 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:30:07.877411 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:07.877413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:07.877497 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:07.877595 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:30:07.877996 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.878568 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.879524 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:07.879800 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:07.879912 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:07.880359 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:30:07.881040 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:30:07.881157 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:07.881257 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 16:30:07.881343 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:30:07.890022 14966 raft_consensus.cc:1261] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Refusing update from remote peer 0082646c9c7a4237a5bc6f1cb1fc6c77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:07.890125 14828 raft_consensus.cc:1261] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Refusing update from remote peer 0082646c9c7a4237a5bc6f1cb1fc6c77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:07.891175 15166 consensus_queue.cc:1035] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:07.891439 15164 consensus_queue.cc:1035] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:07.898947 15171 mvcc.cc:205] Tried to move back new op lower bound from 7333289197105979392 to 7333289196967186432. Current Snapshot: MvccSnapshot[applied={T|T < 7333289197105979392}]
I20260925 16:30:07.902326 15172 mvcc.cc:205] Tried to move back new op lower bound from 7333289197105979392 to 7333289196967186432. Current Snapshot: MvccSnapshot[applied={T|T < 7333289197105979392}]
I20260925 16:30:07.904253 15173 mvcc.cc:205] Tried to move back new op lower bound from 7333289197105979392 to 7333289196967186432. Current Snapshot: MvccSnapshot[applied={T|T < 7333289197105979392}]
I20260925 16:30:07.911546 14682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:07.911854 14682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:07.915390 14682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Sending DeleteTablet for 3 replicas of tablet cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:07.916532 14808 tablet_service.cc:1514] Processing DeleteTablet for tablet cadce69f81e3485f9727ab62a8eb21d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45480
I20260925 16:30:07.916607 14946 tablet_service.cc:1514] Processing DeleteTablet for tablet cadce69f81e3485f9727ab62a8eb21d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40398
I20260925 16:30:07.916606 15078 tablet_service.cc:1514] Processing DeleteTablet for tablet cadce69f81e3485f9727ab62a8eb21d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55398
I20260925 16:30:07.917227 15181 tablet_replica.cc:330] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: stopping tablet replica
I20260925 16:30:07.917331 15183 tablet_replica.cc:330] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: stopping tablet replica
I20260925 16:30:07.917798 15182 tablet_replica.cc:330] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: stopping tablet replica
I20260925 16:30:07.917860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14724
I20260925 16:30:07.917867 15183 raft_consensus.cc:2229] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:07.918094 15182 raft_consensus.cc:2229] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:30:07.918526 15182 raft_consensus.cc:2258] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:07.919092 15183 raft_consensus.cc:2258] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:07.921905 15182 ts_tablet_manager.cc:1905] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:07.922417 15183 ts_tablet_manager.cc:1905] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:07.925630 15182 ts_tablet_manager.cc:1918] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:07.925848 15182 log.cc:1196] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:07.926174 15183 ts_tablet_manager.cc:1918] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:07.926251 15182 ts_tablet_manager.cc:1939] T cadce69f81e3485f9727ab62a8eb21d5 P 0082646c9c7a4237a5bc6f1cb1fc6c77: Deleting consensus metadata
I20260925 16:30:07.926333 15183 log.cc:1196] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:07.926718 15183 ts_tablet_manager.cc:1939] T cadce69f81e3485f9727ab62a8eb21d5 P 8e5ad0a4c9d14e489d08e36ba1189ba9: Deleting consensus metadata
W20260925 16:30:07.927445 14673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39817 (error 108)
I20260925 16:30:07.927886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14877
I20260925 16:30:07.929916 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.929546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39817 (local address 127.0.0.1:45482)
0925 16:30:07.929703 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.929812 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 16:30:07.930088 14673 catalog_manager.cc:4584] TS 691388f799d8497a8b3f5500b22c0e94 (127.0.98.1:39817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cadce69f81e3485f9727ab62a8eb21d5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39817: connect: Connection refused (error 111)
I20260925 16:30:07.930836 14673 catalog_manager.cc:4841] TS 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): tablet cadce69f81e3485f9727ab62a8eb21d5 (table test-table [id=f2100601e4b54acdacfea3f08b5c88a7]) successfully deleted
I20260925 16:30:07.935211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15015
I20260925 16:30:07.935536 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:07.935169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.0.1:40402)
0925 16:30:07.935349 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:07.935427 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 16:30:07.941323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 14653
I20260925 16:30:07.949157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--webserver_interface=127.0.98.62
--webserver_port=32791
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:08.077495 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:08.078008 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:08.078109 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:08.082952 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:08.083052 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:08.083073 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:08.083089 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:08.083104 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:08.090582 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:08.091858 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:08.093006 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:08.103400 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.103400 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.103457 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:08.104611 15185 server_base.cc:1034] running on GCE node
I20260925 16:30:08.105350 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:08.106699 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:08.107925 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790353808107885 us; error 70 us; skew 500 ppm
Sep 25 16:30:08 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:08.111264 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:08.112372 15185 webserver.cc:460] Webserver started at http://127.0.98.62:32791/ using document root <none> and password file <none>
I20260925 16:30:08.113004 15185 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:08.113087 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:08.118532 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:08.121855 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.124338 15185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:08.124548 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "51fc0e88189044a0b69a2ab7ae766cac"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "19371018548ef46a98dcd5ae96602859"
server_key_iv: "22564d5419d62d02be2d8534bd3ef751"
server_key_version: "encryptionkey@0"
I20260925 16:30:08.125227 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:08.145694 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:08.150120 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:08.150365 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:08.159682 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45435
I20260925 16:30:08.160036 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45435 every 8 connection(s)
I20260925 16:30:08.161252 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:08.167719 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:30:08.168506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15185
I20260925 16:30:08.169080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39817
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45435
--webserver_port=36201
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:08.171105 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap starting.
I20260925 16:30:08.194725 15253 log.cc:824] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:08.210757 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:30:08.211277 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac: Bootstrap complete.
I20260925 16:30:08.216251 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:08.216813 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fc0e88189044a0b69a2ab7ae766cac, State: Initialized, Role: FOLLOWER
I20260925 16:30:08.217442 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:08.217592 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:30:08.217669 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:30:08.217821 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:30:08.220685 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:08.221249 15253 leader_election.cc:302] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fc0e88189044a0b69a2ab7ae766cac; no voters:
I20260925 16:30:08.221741 15253 leader_election.cc:288] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 16:30:08.222110 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:30:08.222525 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [term 3 LEADER]: Becoming Leader. State: Replica: 51fc0e88189044a0b69a2ab7ae766cac, State: Running, Role: LEADER
I20260925 16:30:08.223201 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260925 16:30:08.223639 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:08.225795 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "51fc0e88189044a0b69a2ab7ae766cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260925 16:30:08.226008 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:08.226370 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fc0e88189044a0b69a2ab7ae766cac. Latest consensus state: current_term: 3 leader_uuid: "51fc0e88189044a0b69a2ab7ae766cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc0e88189044a0b69a2ab7ae766cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260925 16:30:08.226424 15272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:08.226611 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc0e88189044a0b69a2ab7ae766cac [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:08.234359 15272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:08.235543 15272 catalog_manager.cc:1259] Loaded cluster ID: ff3f731bc8ba42ceaa1f349f8afd1096
I20260925 16:30:08.235642 15272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 16:30:08.237280 15272 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3455d309ad google::LogMessage::Fail() at ??:0
@ 0x7f3455d3388a google::LogMessage::SendToLog() at ??:0
@ 0x7f3455d304de google::LogMessage::Flush() at ??:0
@ 0x7f3455d31a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f346066ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3460670d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f346066d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f34606b1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3457f823d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3457f82c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3457f843f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3457f73512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f34595b76db start_thread at ??:0
@ 0x7f345482e61f clone at ??:0
W20260925 16:30:08.327069 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:08.327409 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:08.327476 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:08.332096 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:08.332193 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:08.332281 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:30:08.341416 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36201
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:08.343087 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:08.344449 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:08.358764 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.358743 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.359288 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:08.359295 15255 server_base.cc:1034] running on GCE node
I20260925 16:30:08.359894 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:08.360607 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:08.364061 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790353808363999 us; error 73 us; skew 500 ppm
Sep 25 16:30:08 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:08.368345 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:30:08.369714 15255 webserver.cc:460] Webserver started at http://127.0.98.1:36201/ using document root <none> and password file <none>
I20260925 16:30:08.370533 15255 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:08.370617 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:08.376688 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 16:30:08.380882 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.383189 15255 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260925 16:30:08.383414 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "691388f799d8497a8b3f5500b22c0e94"
format_stamp: "Formatted at 2026-09-25 16:30:05 on dist-test-slave-31kt"
server_key: "f9fffa3b4a8d8d29d6fbc750aa91bc3e"
server_key_iv: "e2d1c524097053c220a811beced5fa1c"
server_key_version: "encryptionkey@0"
I20260925 16:30:08.384279 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:08.416064 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:08.420437 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:08.420672 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:08.421427 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:08.422959 15311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:30:08.429517 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:30:08.429695 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 16:30:08.429888 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:30:08.432610 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:30:08.432698 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260925 16:30:08.433236 15311 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap starting.
I20260925 16:30:08.445035 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39817
I20260925 16:30:08.445315 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39817 every 8 connection(s)
I20260925 16:30:08.446486 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:30:08.451252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15255
I20260925 16:30:08.451787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38681
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45435
--webserver_port=34761
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:08.459173 15311 log.cc:824] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:08.471055 15311 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:30:08.471580 15311 tablet_bootstrap.cc:492] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Bootstrap complete.
I20260925 16:30:08.473063 15311 ts_tablet_manager.cc:1397] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.008s
I20260925 16:30:08.477315 15311 raft_consensus.cc:348] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:08.478041 15311 raft_consensus.cc:729] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691388f799d8497a8b3f5500b22c0e94, State: Initialized, Role: FOLLOWER
I20260925 16:30:08.478703 15311 consensus_queue.cc:260] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:08.480060 15311 ts_tablet_manager.cc:1428] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
W20260925 16:30:08.524632 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:30:08.449476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:41871)
0925 16:30:08.450394 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.450474 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:08.452149 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:08.523496 (+ 71347us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":281,"threads_started":1}
W20260925 16:30:08.525540 15419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45435 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45435: Client connection negotiation failed: client connection to 127.0.98.62:45435: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:30:08.526648 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.526229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:55029)
0925 16:30:08.526423 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.526541 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 16:30:08.527590 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.527311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:54765)
0925 16:30:08.527452 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.527517 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 16:30:08.527886 15419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:30:08.588564 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:08.588932 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:08.589000 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:08.593694 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:08.593830 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:08.593899 15422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:30:08.600030 15422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34761
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:08.601254 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:08.602468 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:08.616320 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.616364 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:08.616415 15422 server_base.cc:1034] running on GCE node
W20260925 16:30:08.616333 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:08.617162 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:08.617913 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:08.619141 15422 hybrid_clock.cc:648] HybridClock initialized: now 1790353808619107 us; error 58 us; skew 500 ppm
Sep 25 16:30:08 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:08.622328 15422 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:30:08.623425 15422 webserver.cc:460] Webserver started at http://127.0.98.2:34761/ using document root <none> and password file <none>
I20260925 16:30:08.624058 15422 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:08.624125 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:08.628674 15422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:30:08.631541 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.633260 15422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 16:30:08.633442 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "8f561c824da7c020f535c70432858b90"
server_key_iv: "b257017e7ad78f609919362d0e6a8d76"
server_key_version: "encryptionkey@0"
I20260925 16:30:08.634148 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:08.676004 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:08.681819 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:08.682112 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:08.683019 15422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:08.684278 15422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:08.684341 15422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.684412 15422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:08.684441 15422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.697021 15422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38681
I20260925 16:30:08.697211 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38681 every 8 connection(s)
I20260925 16:30:08.698480 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:30:08.703881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15422
I20260925 16:30:08.703975 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.701678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:42453)
0925 16:30:08.702761 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.702954 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":150,"threads_started":1}
I20260925 16:30:08.704396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46677
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45435
--webserver_port=33389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:08.704645 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45435 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45435: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
I20260925 16:30:08.705590 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.705255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:57601)
0925 16:30:08.705422 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.705505 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 16:30:08.706583 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.706236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:38901)
0925 16:30:08.706438 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.706493 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260925 16:30:08.706841 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:30:08.842486 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:08.842891 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:08.842949 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:08.847582 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:08.847695 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:08.847769 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:30:08.854771 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33389
--tserver_master_addrs=127.0.98.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:08.856423 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:08.857836 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:08.870229 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.870216 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:08.870223 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:08.870692 15556 server_base.cc:1034] running on GCE node
I20260925 16:30:08.871191 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:08.871927 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:08.873157 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790353808873126 us; error 56 us; skew 500 ppm
Sep 25 16:30:08 dist-test-slave-31kt krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:08.876801 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:30:08.878300 15556 webserver.cc:460] Webserver started at http://127.0.98.3:33389/ using document root <none> and password file <none>
I20260925 16:30:08.879245 15556 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:08.879352 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:08.885555 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:30:08.889083 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.891530 15556 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:30:08.891748 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77"
format_stamp: "Formatted at 2026-09-25 16:30:06 on dist-test-slave-31kt"
server_key: "7d074605ae4f7bdbf55e955a66e39a03"
server_key_iv: "d798e3015fa2f278e0c8c3b10ad71f63"
server_key_version: "encryptionkey@0"
I20260925 16:30:08.892620 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:08.918867 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:08.923813 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:08.924129 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:08.924999 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:08.926174 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:08.926240 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.926307 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:08.926331 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:08.938377 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46677
I20260925 16:30:08.938403 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46677 every 8 connection(s)
I20260925 16:30:08.939843 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:30:08.944828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15556
I20260925 16:30:08.945382 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.943093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:37347)
0925 16:30:08.944049 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.944232 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":178,"threads_started":1}
W20260925 16:30:08.946251 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45435 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45435: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
I20260925 16:30:08.947117 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.946776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:59653)
0925 16:30:08.946960 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.947021 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 16:30:08.948112 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:08.947680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:59275)
0925 16:30:08.947906 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:08.947997 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 16:30:08.948416 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:30:09.529474 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:09.528719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:33133)
0925 16:30:09.529155 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:09.529297 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":96,"threads_started":1}
I20260925 16:30:09.708374 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:09.707728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:56255)
0925 16:30:09.708128 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:09.708222 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260925 16:30:09.950055 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:09.949347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:32827)
0925 16:30:09.949738 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:09.949892 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20260925 16:30:10.140877 15692 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:10.141223 15692 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:10.142575 15692 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677), 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
Sep 25 16:30:10 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 16:30:10 dist-test-slave-31kt krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 16:30:10.161116 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.143002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38681 (local address 127.0.98.1:35003)
0925 16:30:10.143500 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:10.143521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:10.143758 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:10.145712 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:10.145716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:10.146254 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:10.146877 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:10.146900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.148664 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.148668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:10.149804 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:10.149818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.150695 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.150699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:10.150996 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:10.151812 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:10.151833 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:10.153988 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:10.157883 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:10.157887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:10.157889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:10.158182 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:10.158745 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:10.158749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:10.158751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:10.158894 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:10.159965 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:10.159973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:10.160376 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 16:30:10.160583 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 16:30:10.160941 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":224,"spinlock_wait_cycles":53120,"thread_start_us":146,"threads_started":1}
I20260925 16:30:10.161345 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.143002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46677 (local address 127.0.98.1:37145)
0925 16:30:10.143542 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:10.143558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:10.143758 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:10.145536 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:10.145543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:10.146265 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:10.146877 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:10.146903 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.148903 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.148907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:10.149801 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:10.149819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.150698 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.150703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:10.150974 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:10.151802 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:10.151837 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:10.154358 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:10.157552 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:10.157561 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:10.157582 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:10.158134 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:10.158702 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:10.158706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:10.158708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:10.158913 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:10.159999 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:10.160003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:10.160374 (+ 371us) client_negotiation.cc:769] Sending connection context
0925 16:30:10.160585 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 16:30:10.161075 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":240,"spinlock_wait_cycles":44672,"thread_start_us":114,"threads_started":1}
I20260925 16:30:10.161725 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.143216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37145 (local address 127.0.98.3:46677)
0925 16:30:10.143725 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:10.143761 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:10.143808 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:30:10.144588 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:10.144595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:10.145025 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:10.145373 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:10.147119 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.148706 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.149952 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.150544 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.150727 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:10.154673 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:10.154681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:10.154695 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:10.154741 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:10.157305 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:10.158328 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:10.158337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:10.158344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:10.158540 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:10.159064 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:10.159068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:10.159069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:10.159835 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:10.160007 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:10.160771 (+ 764us) server_negotiation.cc:299] Negotiation successful
0925 16:30:10.161132 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":143,"threads_started":1}
I20260925 16:30:10.162299 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.143216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35003 (local address 127.0.98.2:38681)
0925 16:30:10.144064 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:10.144092 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:10.144139 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:30:10.144986 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:10.144993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:10.145406 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:10.145734 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:10.147104 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.148460 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.149939 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:10.150545 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:10.150735 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:10.154339 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:10.154348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:10.154367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:10.154417 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:10.157303 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:10.158385 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:10.158395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:10.158400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:10.158588 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:10.159073 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:10.159077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:10.159078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:10.159723 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:10.159887 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:10.161683 (+ 1796us) server_negotiation.cc:299] Negotiation successful
0925 16:30:10.162113 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":118,"threads_started":1}
W20260925 16:30:10.164750 15307 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
W20260925 16:30:10.165036 15308 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:10.165191 15308 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 691388f799d8497a8b3f5500b22c0e94; no voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 8e5ad0a4c9d14e489d08e36ba1189ba9
I20260925 16:30:10.165519 15692 raft_consensus.cc:2726] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:30:10.531080 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.530613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:43019)
0925 16:30:10.530883 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:10.530980 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 16:30:10.710016 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.709395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:52991)
0925 16:30:10.709813 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:10.709902 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260925 16:30:10.951726 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:10.951132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:46821)
0925 16:30:10.951498 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:10.951590 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":94,"threads_started":1}
I20260925 16:30:11.532728 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:11.532145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:40353)
0925 16:30:11.532519 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:11.532601 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260925 16:30:11.711745 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:11.711119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:33401)
0925 16:30:11.711514 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:11.711607 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260925 16:30:11.953481 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:11.952802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:34851)
0925 16:30:11.953244 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:11.953334 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":163,"threads_started":1}
I20260925 16:30:12.219944 15702 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:12.220105 15702 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:12.220630 15702 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677), 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
W20260925 16:30:12.221949 15307 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
W20260925 16:30:12.222180 15308 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:12.222245 15308 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 691388f799d8497a8b3f5500b22c0e94; no voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 8e5ad0a4c9d14e489d08e36ba1189ba9
I20260925 16:30:12.222455 15702 raft_consensus.cc:2726] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:30:12.534478 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:12.533839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:54647)
0925 16:30:12.534293 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:12.534375 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
I20260925 16:30:12.713557 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:12.712941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:33963)
0925 16:30:12.713327 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:12.713427 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":88,"threads_started":1}
I20260925 16:30:12.955170 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:12.954542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:35281)
0925 16:30:12.954945 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:12.955034 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
I20260925 16:30:13.536531 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:13.535809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:35053)
0925 16:30:13.536300 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:13.536399 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":114,"threads_started":1}
I20260925 16:30:13.715476 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:13.714846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:42677)
0925 16:30:13.715232 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:13.715340 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
I20260925 16:30:13.938977 15708 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:13.939203 15708 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:13.939798 15708 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677), 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
W20260925 16:30:13.940869 15308 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
W20260925 16:30:13.940979 15307 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:13.941040 15307 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 691388f799d8497a8b3f5500b22c0e94; no voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 8e5ad0a4c9d14e489d08e36ba1189ba9
I20260925 16:30:13.941273 15708 raft_consensus.cc:2726] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:30:13.956955 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:13.956328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:36643)
0925 16:30:13.956722 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:13.956816 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":102,"threads_started":1}
I20260925 16:30:14.538653 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:14.538037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:56453)
0925 16:30:14.538405 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:14.538525 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260925 16:30:14.717127 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:14.716465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:58705)
0925 16:30:14.716892 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:14.716992 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":93,"threads_started":1}
I20260925 16:30:14.958760 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:14.958098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:45893)
0925 16:30:14.958502 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:14.958600 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":90,"threads_started":1}
I20260925 16:30:15.540419 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:15.539853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:49179)
0925 16:30:15.540201 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:15.540290 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
I20260925 16:30:15.718933 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:15.718256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:50795)
0925 16:30:15.718705 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:15.718801 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":91,"threads_started":1}
I20260925 16:30:15.960635 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:15.959940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:45113)
0925 16:30:15.960411 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:15.960512 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":135,"threads_started":1}
I20260925 16:30:16.404160 15716 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:16.404331 15716 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:16.404786 15716 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677), 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
W20260925 16:30:16.405889 15308 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
W20260925 16:30:16.406067 15307 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:16.406183 15307 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 691388f799d8497a8b3f5500b22c0e94; no voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 8e5ad0a4c9d14e489d08e36ba1189ba9
I20260925 16:30:16.406344 15716 raft_consensus.cc:2726] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:30:16.542227 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:16.541513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:41925)
0925 16:30:16.542016 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:16.542116 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":126,"threads_started":1}
I20260925 16:30:16.720680 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:16.720009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:42595)
0925 16:30:16.720469 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:16.720566 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":132,"threads_started":1}
I20260925 16:30:16.962456 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:16.961852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:50557)
0925 16:30:16.962231 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:16.962326 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":89,"threads_started":1}
I20260925 16:30:17.543682 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:17.543173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:47317)
0925 16:30:17.543493 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:17.543578 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
I20260925 16:30:17.722617 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:17.721928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:48809)
0925 16:30:17.722404 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:17.722495 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":139,"threads_started":1}
I20260925 16:30:17.964196 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:17.963460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:38879)
0925 16:30:17.963896 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:17.963988 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":124,"threads_started":1}
I20260925 16:30:18.545500 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:18.544834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:60933)
0925 16:30:18.545272 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:18.545359 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":131,"threads_started":1}
I20260925 16:30:18.724469 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:18.723800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:32931)
0925 16:30:18.724268 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:18.724360 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":126,"threads_started":1}
I20260925 16:30:18.966063 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:18.965369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:52831)
0925 16:30:18.965851 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:18.965945 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":110,"threads_started":1}
I20260925 16:30:19.547210 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:19.546588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:36915)
0925 16:30:19.547019 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:19.547106 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260925 16:30:19.726063 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:19.725529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:52569)
0925 16:30:19.725855 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:19.725943 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260925 16:30:19.967805 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:19.967133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:48311)
0925 16:30:19.967604 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:19.967690 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":184,"threads_started":1}
I20260925 16:30:20.548712 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:20.548187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:37923)
0925 16:30:20.548524 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:20.548589 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":139,"threads_started":1}
I20260925 16:30:20.727833 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:20.727190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:37669)
0925 16:30:20.727558 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:20.727657 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
I20260925 16:30:20.969574 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:20.969014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:57517)
0925 16:30:20.969363 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:20.969451 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260925 16:30:21.550825 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:21.550216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:59117)
0925 16:30:21.550561 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:21.550689 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260925 16:30:21.729699 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:21.729009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:37947)
0925 16:30:21.729452 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:21.729555 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":95,"threads_started":1}
I20260925 16:30:21.971379 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:21.970699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:44269)
0925 16:30:21.971153 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:21.971246 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":111,"threads_started":1}
I20260925 16:30:22.552412 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:22.551949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:53439)
0925 16:30:22.552236 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:22.552305 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260925 16:30:22.731406 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:22.730814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:60653)
0925 16:30:22.731176 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:22.731268 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260925 16:30:22.972971 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:22.972483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.3:41209)
0925 16:30:22.972799 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:22.972864 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":128,"threads_started":1}
I20260925 16:30:23.426484 15738 raft_consensus.cc:482] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:23.426719 15738 raft_consensus.cc:504] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0082646c9c7a4237a5bc6f1cb1fc6c77" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46677 } } peers { permanent_uuid: "8e5ad0a4c9d14e489d08e36ba1189ba9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38681 } } peers { permanent_uuid: "691388f799d8497a8b3f5500b22c0e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39817 } }
I20260925 16:30:23.427268 15738 leader_election.cc:288] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677), 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681)
W20260925 16:30:23.428541 15308 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0082646c9c7a4237a5bc6f1cb1fc6c77 (127.0.98.3:46677): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
W20260925 16:30:23.428721 15307 leader_election.cc:341] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8e5ad0a4c9d14e489d08e36ba1189ba9 (127.0.98.2:38681): Not found: Tablet not found: cadce69f81e3485f9727ab62a8eb21d5
I20260925 16:30:23.428807 15307 leader_election.cc:302] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 691388f799d8497a8b3f5500b22c0e94; no voters: 0082646c9c7a4237a5bc6f1cb1fc6c77, 8e5ad0a4c9d14e489d08e36ba1189ba9
I20260925 16:30:23.429021 15738 raft_consensus.cc:2726] T cadce69f81e3485f9727ab62a8eb21d5 P 691388f799d8497a8b3f5500b22c0e94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:30:23.554239 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:23.553644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.1:35607)
0925 16:30:23.554030 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:23.554097 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260925 16:30:23.733342 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:23.732720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45435 (local address 127.0.98.2:49909)
0925 16:30:23.733112 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:23.733205 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":95,"threads_started":1}
I20260925 16:30:23.951761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15255
I20260925 16:30:23.959065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15422
I20260925 16:30:23.965586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15556
2026-09-25T16:30:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19784 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:30:24 dist-test-slave-31kt krb5kdc[15743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:30:24 dist-test-slave-31kt krb5kdc[15743](info): set up 2 sockets
Sep 25 16:30:24 dist-test-slave-31kt krb5kdc[15743](info): commencing operation
krb5kdc: starting...
W20260925 16:30:24.958128 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:30:25 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:30:25.107334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:25.254544 15756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:25.254984 15756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:25.255051 15756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:25.259881 15756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:30:25.259999 15756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:30:25.260025 15756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:30:25.260043 15756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:25.260071 15756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:25.260095 15756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:25.260118 15756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:25.260149 15756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:25.267585 15756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:25.269228 15756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:25.270776 15756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:30:25.282987 15756 server_base.cc:1034] running on GCE node
W20260925 16:30:25.282863 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:25.282863 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:25.282863 15764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:25.284130 15756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:25.286072 15756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:25.287365 15756 hybrid_clock.cc:648] HybridClock initialized: now 1790353825287359 us; error 74 us; skew 500 ppm
Sep 25 16:30:25 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:25.295737 15756 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:25.297039 15756 webserver.cc:460] Webserver started at http://127.0.98.62:44333/ using document root <none> and password file <none>
I20260925 16:30:25.297727 15756 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:25.297857 15756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:25.298102 15756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:25.300513 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "c9194492681e48788a4c42a52436f501"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "c9fc6a926c51542f0f7f70e337802d59"
server_key_iv: "527ca60182c06b7217e16d374bff1752"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.301137 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9194492681e48788a4c42a52436f501"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "c9fc6a926c51542f0f7f70e337802d59"
server_key_iv: "527ca60182c06b7217e16d374bff1752"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.306592 15756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 16:30:25.310359 15771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:25.312371 15756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:30:25.312583 15756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "c9194492681e48788a4c42a52436f501"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "c9fc6a926c51542f0f7f70e337802d59"
server_key_iv: "527ca60182c06b7217e16d374bff1752"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.312737 15756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:25.333645 15756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:25.337598 15756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:25.337854 15756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:25.346026 15756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33071
I20260925 16:30:25.346356 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33071 every 8 connection(s)
I20260925 16:30:25.347425 15756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:25.347842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15756
I20260925 16:30:25.347997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:30:25.348385 392 external_mini_cluster.cc:1428] Setting key e3d640b8467b7e0525555ac91daa0773
I20260925 16:30:25.351390 15824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:25.359388 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap starting.
Sep 25 16:30:25 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:25.362251 15824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:25.363202 15824 log.cc:824] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:25.366086 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: No bootstrap required, opened a new log
I20260925 16:30:25.367674 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:25.350384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.0.98.62:33071)
0925 16:30:25.350990 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:25.351015 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:25.351068 (+ 53us) server_negotiation.cc:407] Connection header received
0925 16:30:25.351978 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:25.352004 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:25.352485 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:25.352912 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:25.353991 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:25.355280 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:25.356733 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:25.357491 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:25.357685 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:25.361233 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:25.361242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:25.361265 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:25.361322 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:25.363786 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:25.364436 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:25.364444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:25.364451 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:25.364606 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:25.364951 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:25.364954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:25.364955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:25.365581 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:25.365808 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:25.366232 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 16:30:25.366658 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":194,"threads_started":1}
I20260925 16:30:25.370518 15824 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:25.370850 15824 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:25.370911 15824 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9194492681e48788a4c42a52436f501, State: Initialized, Role: FOLLOWER
I20260925 16:30:25.371420 15824 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:25.371768 15824 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:30:25.371876 15824 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:30:25.372030 15824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:25.373433 15824 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:25.374092 15824 leader_election.cc:302] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9194492681e48788a4c42a52436f501; no voters:
I20260925 16:30:25.374643 15824 leader_election.cc:288] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:30:25.374806 15829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:25.375124 15829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 LEADER]: Becoming Leader. State: Replica: c9194492681e48788a4c42a52436f501, State: Running, Role: LEADER
I20260925 16:30:25.375582 15829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:25.376098 15824 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:25.377540 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9194492681e48788a4c42a52436f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260925 16:30:25.377802 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:25.378268 15841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:25.378666 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9194492681e48788a4c42a52436f501. Latest consensus state: current_term: 1 leader_uuid: "c9194492681e48788a4c42a52436f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260925 16:30:25.379016 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:25.382588 15841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:25.390605 15841 catalog_manager.cc:1347] Generated new cluster ID: f4531ea0080e407fb172eaca91a58f44
I20260925 16:30:25.390740 15841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:30:25.415336 15841 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:30:25.416884 15841 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:30:25.425276 15841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Generated new TSK 0
I20260925 16:30:25.426554 15841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:30:25.478595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33071
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:25.628551 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:25.628930 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:25.628983 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:25.634747 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:25.634877 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:25.634954 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:30:25.642067 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:25.643356 15850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:25.644903 15850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:25.658623 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:25.658854 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:25.659229 15850 server_base.cc:1034] running on GCE node
W20260925 16:30:25.658798 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:25.659850 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:25.660677 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:25.661867 15850 hybrid_clock.cc:648] HybridClock initialized: now 1790353825661840 us; error 51 us; skew 500 ppm
Sep 25 16:30:25 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:25.665714 15850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:30:25.666885 15850 webserver.cc:460] Webserver started at http://127.0.98.1:42009/ using document root <none> and password file <none>
I20260925 16:30:25.667500 15850 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:25.667577 15850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:25.667819 15850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:25.670083 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "afd3d45619e1486a919c25c8a9b4c5de"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "f37afd725c712539b670c8a29ae721a6"
server_key_iv: "d51fc00925c3813bd0b436cb8a3a14b3"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.670658 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "afd3d45619e1486a919c25c8a9b4c5de"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "f37afd725c712539b670c8a29ae721a6"
server_key_iv: "d51fc00925c3813bd0b436cb8a3a14b3"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.675271 15850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:30:25.678624 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:25.680739 15850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:25.680915 15850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "afd3d45619e1486a919c25c8a9b4c5de"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "f37afd725c712539b670c8a29ae721a6"
server_key_iv: "d51fc00925c3813bd0b436cb8a3a14b3"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.681073 15850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:25.702569 15850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:25.707026 15850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:25.707245 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:25.707952 15850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:25.709014 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:25.709556 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260925 16:30:25.709612 15850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:25.709628 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:25.720655 15850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39747
I20260925 16:30:25.720734 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39747 every 8 connection(s)
I20260925 16:30:25.721735 15850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:30:25.729370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15850
I20260925 16:30:25.729527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:30:25.730115 392 external_mini_cluster.cc:1428] Setting key d950d758765b0f139c5ae288b0cd0b8c
Sep 25 16:30:25 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:30:25.741282 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:25.724727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40585 (local address 127.0.98.62:33071)
0925 16:30:25.724925 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:25.724929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:25.726063 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 16:30:25.727778 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:25.727782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:25.727863 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:25.727949 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:25.729775 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:25.730468 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:25.732093 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:25.732362 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:25.732459 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:25.736263 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:25.736268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:25.736271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:25.736311 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:25.738066 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:25.739131 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:25.739144 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:25.739146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:25.739254 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:25.739749 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:25.739752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:25.739753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:25.739992 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:25.740085 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:25.740953 (+ 868us) server_negotiation.cc:299] Negotiation successful
0925 16:30:25.741117 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 16:30:25.742588 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:25.725094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:40585)
0925 16:30:25.725841 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:25.725911 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:25.727466 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:25.728212 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:25.728223 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:25.728773 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:25.729537 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:25.729567 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:25.730641 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:25.730645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:25.731962 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:25.731978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:25.732578 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:25.732586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:25.732803 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:25.733679 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:25.733702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:25.735897 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:25.738326 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:25.738340 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:25.738373 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:25.738946 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:25.739367 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:25.739371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:25.739373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:25.739636 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:25.740114 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:25.740126 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:25.740622 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 16:30:25.740915 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 16:30:25.741607 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":153,"threads_started":1}
I20260925 16:30:25.744653 15979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:30:25.745069 15979 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:25.745922 15979 heartbeater.cc:511] Master 127.0.98.62:33071 requested a full tablet report, sending...
I20260925 16:30:25.748522 15786 ts_manager.cc:194] Registered new tserver with Master: afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747)
I20260925 16:30:25.750275 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:30:25.764847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33071
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:25.908737 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:25.909098 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:25.909155 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:25.913504 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:25.913589 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:25.913653 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:30:25.919754 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:25.920939 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:25.922052 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:25.934665 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:25.934665 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:25.934707 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:25.935045 15984 server_base.cc:1034] running on GCE node
I20260925 16:30:25.935806 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:25.936721 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:25.937960 15984 hybrid_clock.cc:648] HybridClock initialized: now 1790353825937937 us; error 50 us; skew 500 ppm
Sep 25 16:30:25 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:25.941604 15984 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:30:25.942835 15984 webserver.cc:460] Webserver started at http://127.0.98.2:39083/ using document root <none> and password file <none>
I20260925 16:30:25.943553 15984 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:25.943621 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:25.943856 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:25.946827 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "20d78789c37b42d3b416ae63867ad677"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "22498727f88bd59adb7b35ab36c41d89"
server_key_iv: "9a27cd9f1b8409c340d70a27b02993d7"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.947726 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20d78789c37b42d3b416ae63867ad677"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "22498727f88bd59adb7b35ab36c41d89"
server_key_iv: "9a27cd9f1b8409c340d70a27b02993d7"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.953393 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:30:25.957296 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:25.959311 15984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:30:25.959492 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "20d78789c37b42d3b416ae63867ad677"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "22498727f88bd59adb7b35ab36c41d89"
server_key_iv: "9a27cd9f1b8409c340d70a27b02993d7"
server_key_version: "encryptionkey@0"
I20260925 16:30:25.959681 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:25.994524 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:25.999233 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:25.999495 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:26.000276 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:26.001345 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:26.001407 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:26.001472 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:26.001499 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:26.013969 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42909
I20260925 16:30:26.013993 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42909 every 8 connection(s)
I20260925 16:30:26.015257 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:30:26.025542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 15984
I20260925 16:30:26.025677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:30:26.025990 392 external_mini_cluster.cc:1428] Setting key 0863ad0dd2a1ffb0f1511f811cee37a3
Sep 25 16:30:26 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:26.034312 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.017615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48687 (local address 127.0.98.62:33071)
0925 16:30:26.017829 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.017834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.019046 (+ 1212us) server_negotiation.cc:407] Connection header received
0925 16:30:26.020701 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.020706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.020789 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.020876 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:26.022689 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.023314 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.024875 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.025158 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.025267 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.028963 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:26.028968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:26.028971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:26.029011 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:26.030920 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:26.031838 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:26.031842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:26.031844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:26.031969 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:26.032518 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:26.032522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:26.032524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:26.032808 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:26.032950 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.033894 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.034092 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 16:30:26.035545 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.018044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:48687)
0925 16:30:26.018807 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.018877 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.020397 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.021148 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.021159 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.021682 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:26.022459 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.022479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.023469 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.023473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.024689 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.024703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.025411 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.025418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.025632 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.026469 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:26.026491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:26.028688 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:26.031113 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:26.031123 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:26.031149 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:26.031655 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:26.032113 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:26.032116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:26.032117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:26.032331 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:26.032988 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:26.032997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:26.033452 (+ 455us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.033799 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.034224 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":173,"threads_started":1}
I20260925 16:30:26.037254 16113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33071
I20260925 16:30:26.037551 16113 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:30:26.038267 16113 heartbeater.cc:511] Master 127.0.98.62:33071 requested a full tablet report, sending...
I20260925 16:30:26.040104 15786 ts_manager.cc:194] Registered new tserver with Master: 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909)
I20260925 16:30:26.040891 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:30:26.067911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33071
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:26.198381 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:26.198789 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:26.199033 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:26.204723 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:26.204867 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:26.204950 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:30:26.212365 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:26.213666 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:26.214836 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:26.226987 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:26.227035 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:26.227058 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:26.227376 16118 server_base.cc:1034] running on GCE node
I20260925 16:30:26.228236 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:26.229169 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:26.230520 16118 hybrid_clock.cc:648] HybridClock initialized: now 1790353826230491 us; error 40 us; skew 500 ppm
Sep 25 16:30:26 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:26.233896 16118 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:30:26.235033 16118 webserver.cc:460] Webserver started at http://127.0.98.3:38547/ using document root <none> and password file <none>
I20260925 16:30:26.235679 16118 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:26.235761 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:26.235976 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:30:26.239548 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "d43cb969035c456c93e56891aa471a3a"
format_stamp: "Formatted at 2026-09-25 16:30:26 on dist-test-slave-31kt"
server_key: "350ca0187fffa6fa0d09672c5618947b"
server_key_iv: "ed26c8c5aaeaf59e5e93cafa2d71743f"
server_key_version: "encryptionkey@0"
I20260925 16:30:26.240250 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d43cb969035c456c93e56891aa471a3a"
format_stamp: "Formatted at 2026-09-25 16:30:26 on dist-test-slave-31kt"
server_key: "350ca0187fffa6fa0d09672c5618947b"
server_key_iv: "ed26c8c5aaeaf59e5e93cafa2d71743f"
server_key_version: "encryptionkey@0"
I20260925 16:30:26.245055 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:30:26.248732 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:26.250730 16118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:30:26.250913 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "d43cb969035c456c93e56891aa471a3a"
format_stamp: "Formatted at 2026-09-25 16:30:26 on dist-test-slave-31kt"
server_key: "350ca0187fffa6fa0d09672c5618947b"
server_key_iv: "ed26c8c5aaeaf59e5e93cafa2d71743f"
server_key_version: "encryptionkey@0"
I20260925 16:30:26.251055 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:26.267051 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:26.271512 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:26.271764 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:26.272552 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:26.273931 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:26.274014 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:26.274083 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:26.274112 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:26.285724 16118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43495
I20260925 16:30:26.285746 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43495 every 8 connection(s)
I20260925 16:30:26.286973 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:30:26.287226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16118
I20260925 16:30:26.287395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:30:26.287765 392 external_mini_cluster.cc:1428] Setting key 1f268a3255d58cd027234d067c32be51
Sep 25 16:30:26 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:26.305832 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.289045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56341 (local address 127.0.98.62:33071)
0925 16:30:26.289237 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.289243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.290905 (+ 1662us) server_negotiation.cc:407] Connection header received
0925 16:30:26.292182 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.292185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.292257 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.292371 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:26.294359 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.294964 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.296442 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.296717 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.296842 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.300587 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:26.300598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:26.300601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:26.300643 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:26.302633 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:26.303653 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:26.303655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:26.303657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:26.303844 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:26.304339 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:26.304341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:26.304342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:26.304604 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:26.304699 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.305439 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.305626 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 16:30:26.307024 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.289575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:56341)
0925 16:30:26.290707 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.290777 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.291834 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.292739 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.292753 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.293302 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:26.294110 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.294131 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.295148 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.295152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.296251 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.296264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.296911 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.296914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.297125 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.298160 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:26.298184 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:26.300238 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:26.302921 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:26.302931 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:26.302954 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:26.303482 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:26.303981 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:26.303984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:26.303986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:26.304211 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:26.304722 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:26.304729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:26.305120 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.305332 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.305734 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":191,"threads_started":1}
I20260925 16:30:26.308913 16247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33071
I20260925 16:30:26.309253 16247 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:26.309921 16247 heartbeater.cc:511] Master 127.0.98.62:33071 requested a full tablet report, sending...
I20260925 16:30:26.311395 15786 ts_manager.cc:194] Registered new tserver with Master: d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
I20260925 16:30:26.312156 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56341
I20260925 16:30:26.316496 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:30:26.328745 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.319355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.0.98.62:33071)
0925 16:30:26.319498 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.319502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.319674 (+ 172us) server_negotiation.cc:407] Connection header received
0925 16:30:26.319782 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.319785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.319840 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.319950 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:26.320790 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.321374 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.322566 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.322829 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.322959 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.324658 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:26.324662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:26.324664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:26.324702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:26.326522 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:26.327061 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:26.327064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:26.327065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:26.327213 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:26.327560 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:26.327563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:26.327564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:26.327852 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:26.327994 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.328407 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.328563 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:30:26.332894 15786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:30:26.337555 15786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:30:26.357297 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.350066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43495 (local address 127.0.0.1:45768)
0925 16:30:26.350858 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.350871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.350954 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.351817 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.351821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.351841 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:26.351978 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.351985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.353451 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.353454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.354916 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.354925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.356620 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.356624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.356757 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.356798 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.356924 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.357103 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":120,"threads_started":1}
I20260925 16:30:26.357563 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.349807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39747 (local address 127.0.0.1:47696)
0925 16:30:26.350538 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.350552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.350649 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.351502 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.351509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.351539 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:26.351694 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.351704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.353287 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.353291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.355301 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.355319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.357042 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.357047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.357158 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.357188 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.357263 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.357370 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":125,"threads_started":1}
I20260925 16:30:26.358268 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.350702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.0.98.3:43495)
0925 16:30:26.351210 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.351230 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.351255 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:30:26.351340 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.351344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.351492 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.351628 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:26.352123 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.353279 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.355121 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.356126 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.356353 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.356557 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.357958 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.358106 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":159,"threads_started":1}
I20260925 16:30:26.358906 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.350287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.98.1:39747)
0925 16:30:26.350702 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.350710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.350746 (+ 36us) server_negotiation.cc:407] Connection header received
0925 16:30:26.350845 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.350851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.351094 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.351284 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:26.351845 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.353141 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.355516 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.356668 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.356786 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.356962 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.357696 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.358745 (+ 1049us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20260925 16:30:26.358953 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.348700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.0.98.2:42909)
0925 16:30:26.349384 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.349400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.349929 (+ 529us) server_negotiation.cc:407] Connection header received
0925 16:30:26.351135 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.351143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.351440 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.351600 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:26.353157 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.354394 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.357067 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.358024 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.358108 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.358263 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.358671 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.358787 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":185,"threads_started":1}
I20260925 16:30:26.359333 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.348492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42909 (local address 127.0.0.1:39636)
0925 16:30:26.349719 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.349824 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.350070 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.352905 (+ 2835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.352908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.352927 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:26.353033 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.353038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.354579 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.354582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.356935 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.356943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.358380 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.358384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.358502 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.358528 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.359104 (+ 576us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.359205 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":96,"threads_started":1}
I20260925 16:30:26.362452 16181 tablet_service.cc:1467] Processing CreateTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 (DEFAULT_TABLE table=test-table [id=1887e8959ceb438f93dd3a2309be0d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:26.362910 15913 tablet_service.cc:1467] Processing CreateTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 (DEFAULT_TABLE table=test-table [id=1887e8959ceb438f93dd3a2309be0d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:26.363806 16181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe54dc5053e74468b5a10fa542c3c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:26.364531 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe54dc5053e74468b5a10fa542c3c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:26.364032 16045 tablet_service.cc:1467] Processing CreateTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 (DEFAULT_TABLE table=test-table [id=1887e8959ceb438f93dd3a2309be0d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:26.365173 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe54dc5053e74468b5a10fa542c3c7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:26.372725 16264 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Bootstrap starting.
I20260925 16:30:26.374197 16266 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Bootstrap starting.
I20260925 16:30:26.374231 16265 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap starting.
I20260925 16:30:26.376091 16264 tablet_bootstrap.cc:654] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:26.376781 16265 tablet_bootstrap.cc:654] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:26.377200 16266 tablet_bootstrap.cc:654] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:26.377884 16265 log.cc:824] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:26.377849 16264 log.cc:824] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:26.378199 16266 log.cc:824] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:26.380604 16266 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: No bootstrap required, opened a new log
I20260925 16:30:26.380705 16264 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: No bootstrap required, opened a new log
I20260925 16:30:26.380829 16266 ts_tablet_manager.cc:1397] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:30:26.380947 16264 ts_tablet_manager.cc:1397] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 16:30:26.383580 16265 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: No bootstrap required, opened a new log
I20260925 16:30:26.383917 16265 ts_tablet_manager.cc:1397] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 16:30:26.384598 16264 raft_consensus.cc:348] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.384924 16264 raft_consensus.cc:374] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:26.384999 16264 raft_consensus.cc:729] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43cb969035c456c93e56891aa471a3a, State: Initialized, Role: FOLLOWER
I20260925 16:30:26.385002 16266 raft_consensus.cc:348] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.385386 16266 raft_consensus.cc:374] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:26.385488 16266 raft_consensus.cc:729] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d78789c37b42d3b416ae63867ad677, State: Initialized, Role: FOLLOWER
I20260925 16:30:26.385624 16264 consensus_queue.cc:260] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.386093 16266 consensus_queue.cc:260] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.387065 16247 heartbeater.cc:503] Master 127.0.98.62:33071 was elected leader, sending a full tablet report...
I20260925 16:30:26.387228 16264 ts_tablet_manager.cc:1428] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:30:26.387547 16266 ts_tablet_manager.cc:1428] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:30:26.388010 16113 heartbeater.cc:503] Master 127.0.98.62:33071 was elected leader, sending a full tablet report...
I20260925 16:30:26.389219 16265 raft_consensus.cc:348] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.389539 16265 raft_consensus.cc:374] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:26.389662 16265 raft_consensus.cc:729] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd3d45619e1486a919c25c8a9b4c5de, State: Initialized, Role: FOLLOWER
I20260925 16:30:26.389926 16270 raft_consensus.cc:482] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:26.390153 16270 raft_consensus.cc:504] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.390295 16265 consensus_queue.cc:260] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.391501 16265 ts_tablet_manager.cc:1428] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:30:26.392333 15979 heartbeater.cc:503] Master 127.0.98.62:33071 was elected leader, sending a full tablet report...
I20260925 16:30:26.392829 16270 leader_election.cc:288] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747)
I20260925 16:30:26.396757 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.391785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42909 (local address 127.0.98.3:58403)
0925 16:30:26.392024 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.392038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.392157 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.392539 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.392542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.392562 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:26.392704 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.392711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.393882 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.393886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.395222 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.395232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.396231 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.396235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.396360 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.396399 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.396482 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.396582 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":5}
I20260925 16:30:26.396839 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.391891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58403 (local address 127.0.98.2:42909)
0925 16:30:26.392026 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.392031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.392087 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:30:26.392276 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.392279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.392330 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.392402 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:26.392838 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.393547 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.395366 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.395970 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.396077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.396163 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.396529 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.396661 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:30:26.397610 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe54dc5053e74468b5a10fa542c3c7b2" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d78789c37b42d3b416ae63867ad677" is_pre_election: true
I20260925 16:30:26.397842 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.392712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43997 (local address 127.0.98.1:39747)
0925 16:30:26.392845 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.392850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.393287 (+ 437us) server_negotiation.cc:407] Connection header received
0925 16:30:26.393417 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.393420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.393473 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.393553 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:26.394104 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.395045 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.396559 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.397223 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.397324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.397397 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.397581 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.397676 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:30:26.397747 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.392586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39747 (local address 127.0.98.3:43997)
0925 16:30:26.393176 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.393191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:26.393295 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:26.393691 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:26.393696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:26.393719 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:26.393945 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.393958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.395210 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.395213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.396410 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:26.396420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.397347 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.397350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:26.397443 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.397474 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:30:26.397544 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:30:26.397616 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":126,"threads_started":1}
I20260925 16:30:26.398140 16067 raft_consensus.cc:2454] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 0.
I20260925 16:30:26.398756 16134 leader_election.cc:302] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a; no voters:
I20260925 16:30:26.398838 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe54dc5053e74468b5a10fa542c3c7b2" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd3d45619e1486a919c25c8a9b4c5de" is_pre_election: true
I20260925 16:30:26.399148 16272 raft_consensus.cc:2781] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:30:26.399235 15933 raft_consensus.cc:2454] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 0.
I20260925 16:30:26.399279 16272 raft_consensus.cc:482] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:30:26.399333 16272 raft_consensus.cc:3037] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:26.400862 16272 raft_consensus.cc:504] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.401314 16272 leader_election.cc:288] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 election: Requested vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747)
I20260925 16:30:26.401744 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe54dc5053e74468b5a10fa542c3c7b2" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d78789c37b42d3b416ae63867ad677"
I20260925 16:30:26.401949 16067 raft_consensus.cc:3037] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:26.401916 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe54dc5053e74468b5a10fa542c3c7b2" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd3d45619e1486a919c25c8a9b4c5de"
I20260925 16:30:26.402192 15933 raft_consensus.cc:3037] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:26.403313 16067 raft_consensus.cc:2454] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 1.
I20260925 16:30:26.403476 15933 raft_consensus.cc:2454] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 1.
I20260925 16:30:26.403832 16134 leader_election.cc:302] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a; no voters:
I20260925 16:30:26.404100 16272 raft_consensus.cc:2781] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:26.404229 16272 raft_consensus.cc:686] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 1 LEADER]: Becoming Leader. State: Replica: d43cb969035c456c93e56891aa471a3a, State: Running, Role: LEADER
I20260925 16:30:26.404690 16272 consensus_queue.cc:237] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:26.409403 15786 catalog_manager.cc:5495] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to d43cb969035c456c93e56891aa471a3a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d43cb969035c456c93e56891aa471a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } health_report { overall_health: HEALTHY } } }
I20260925 16:30:26.426190 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.421044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.0.98.3:43495)
0925 16:30:26.421226 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:26.421230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:26.421251 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:30:26.421367 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:26.421370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:26.421491 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:26.421612 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:30:26.422142 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.422958 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.423954 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:26.424188 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:26.424279 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:26.424723 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:30:26.425527 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:30:26.425678 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:26.425885 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 16:30:26.425984 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 16:30:26.433969 15933 raft_consensus.cc:1261] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Refusing update from remote peer d43cb969035c456c93e56891aa471a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:26.434216 16067 raft_consensus.cc:1261] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Refusing update from remote peer d43cb969035c456c93e56891aa471a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:26.434757 16272 consensus_queue.cc:1035] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:26.434993 16270 consensus_queue.cc:1035] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:26.446815 16278 mvcc.cc:205] Tried to move back new op lower bound from 7333289273064460288 to 7333289272957554688. Current Snapshot: MvccSnapshot[applied={T|T < 7333289273064460288}]
I20260925 16:30:26.446879 16279 mvcc.cc:205] Tried to move back new op lower bound from 7333289273064460288 to 7333289272957554688. Current Snapshot: MvccSnapshot[applied={T|T < 7333289273064460288}]
I20260925 16:30:26.458420 15786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:26.458659 15786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:26.461648 15786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Sending DeleteTablet for 3 replicas of tablet fe54dc5053e74468b5a10fa542c3c7b2
I20260925 16:30:26.462791 16045 tablet_service.cc:1514] Processing DeleteTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39636
I20260925 16:30:26.462975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15850
I20260925 16:30:26.463042 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47696
I20260925 16:30:26.464182 16287 tablet_replica.cc:330] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: stopping tablet replica
I20260925 16:30:26.464643 16287 raft_consensus.cc:2229] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:26.465308 16287 raft_consensus.cc:2258] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:26.466204 16181 tablet_service.cc:1514] Processing DeleteTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260925 16:30:26.467664 16287 ts_tablet_manager.cc:1905] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:26.470155 16288 tablet_replica.cc:330] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: stopping tablet replica
I20260925 16:30:26.470780 16288 raft_consensus.cc:2229] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:30:26.471081 16287 ts_tablet_manager.cc:1918] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:26.471267 16287 log.cc:1196] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/fe54dc5053e74468b5a10fa542c3c7b2
I20260925 16:30:26.471514 16288 raft_consensus.cc:2258] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:26.471715 16287 ts_tablet_manager.cc:1939] T fe54dc5053e74468b5a10fa542c3c7b2 P 20d78789c37b42d3b416ae63867ad677: Deleting consensus metadata
I20260925 16:30:26.473629 16288 ts_tablet_manager.cc:1905] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:30:26.473989 15775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39747 (error 108)
I20260925 16:30:26.474395 15772 catalog_manager.cc:4841] TS 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): tablet fe54dc5053e74468b5a10fa542c3c7b2 (table test-table [id=1887e8959ceb438f93dd3a2309be0d59]) successfully deleted
I20260925 16:30:26.475150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15984
I20260925 16:30:26.476258 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:26.475224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39747 (local address 127.0.0.1:47708)
0925 16:30:26.476025 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:26.476127 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668}
W20260925 16:30:26.477983 15775 catalog_manager.cc:4584] TS afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe54dc5053e74468b5a10fa542c3c7b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39747: connect: Connection refused (error 111)
I20260925 16:30:26.480825 16288 ts_tablet_manager.cc:1918] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:26.480991 16288 log.cc:1196] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/fe54dc5053e74468b5a10fa542c3c7b2
I20260925 16:30:26.481339 16288 ts_tablet_manager.cc:1939] T fe54dc5053e74468b5a10fa542c3c7b2 P d43cb969035c456c93e56891aa471a3a: Deleting consensus metadata
I20260925 16:30:26.483426 15772 catalog_manager.cc:4841] TS d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): tablet fe54dc5053e74468b5a10fa542c3c7b2 (table test-table [id=1887e8959ceb438f93dd3a2309be0d59]) successfully deleted
I20260925 16:30:26.484866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 16118
I20260925 16:30:26.492347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 15756
I20260925 16:30:46.500849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--webserver_interface=127.0.98.62
--webserver_port=44333
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:30:46.634824 16290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:46.635221 16290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:46.635284 16290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:46.639771 16290 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:30:46.639846 16290 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:30:46.639869 16290 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:30:46.639889 16290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:30:46.639914 16290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:46.639940 16290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:46.639971 16290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:30:46.639997 16290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:30:46.646620 16290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:46.648171 16290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:46.649475 16290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:46.667093 16297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:46.667116 16296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:46.667186 16290 server_base.cc:1034] running on GCE node
W20260925 16:30:46.667120 16299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:46.668283 16290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:46.669974 16290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:46.671193 16290 hybrid_clock.cc:648] HybridClock initialized: now 1790353846671182 us; error 48 us; skew 500 ppm
Sep 25 16:30:46 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:46.679176 16290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:30:46.680835 16290 webserver.cc:460] Webserver started at http://127.0.98.62:44333/ using document root <none> and password file <none>
I20260925 16:30:46.681907 16290 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:46.682052 16290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:46.688164 16290 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:30:46.691974 16306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:46.693856 16290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:30:46.694082 16290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "c9194492681e48788a4c42a52436f501"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "c9fc6a926c51542f0f7f70e337802d59"
server_key_iv: "527ca60182c06b7217e16d374bff1752"
server_key_version: "encryptionkey@0"
I20260925 16:30:46.694901 16290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:46.716316 16290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:46.721112 16290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:46.721385 16290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:46.731091 16290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33071
I20260925 16:30:46.731518 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33071 every 8 connection(s)
I20260925 16:30:46.732702 16290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:30:46.739295 16359 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:30:46.741209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16290
I20260925 16:30:46.741779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39747
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33071
--webserver_port=42009
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:46.742154 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap starting.
I20260925 16:30:46.760787 16359 log.cc:824] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:46.772528 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:30:46.773000 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap complete.
I20260925 16:30:46.777334 16359 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:46.778142 16359 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9194492681e48788a4c42a52436f501, State: Initialized, Role: FOLLOWER
I20260925 16:30:46.778904 16359 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:46.779119 16359 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:30:46.779207 16359 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:30:46.779311 16359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 1 FOLLOWER]: Advancing to term 2
I20260925 16:30:46.781919 16359 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:46.782630 16359 leader_election.cc:302] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9194492681e48788a4c42a52436f501; no voters:
I20260925 16:30:46.783072 16359 leader_election.cc:288] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 16:30:46.783210 16363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 16:30:46.783514 16363 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 LEADER]: Becoming Leader. State: Replica: c9194492681e48788a4c42a52436f501, State: Running, Role: LEADER
I20260925 16:30:46.784027 16363 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:30:46.784648 16359 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:30:46.785584 16364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9194492681e48788a4c42a52436f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260925 16:30:46.785796 16364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:46.786127 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9194492681e48788a4c42a52436f501. Latest consensus state: current_term: 2 leader_uuid: "c9194492681e48788a4c42a52436f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260925 16:30:46.786244 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: This master's current role is: LEADER
I20260925 16:30:46.791213 16379 catalog_manager.cc:1259] Loaded cluster ID: f4531ea0080e407fb172eaca91a58f44
I20260925 16:30:46.791320 16379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: loading cluster ID for follower catalog manager: success
I20260925 16:30:46.794601 16379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: acquiring CA information for follower catalog manager: success
I20260925 16:30:46.796648 16379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:30:46.796846 16372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:30:46.799865 16372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:30:46.800427 16372 catalog_manager.cc:1259] Loaded cluster ID: f4531ea0080e407fb172eaca91a58f44
I20260925 16:30:46.800487 16372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:30:46.801236 16372 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:30:46.801730 16372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Loaded TSK: 0
I20260925 16:30:46.820814 16372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Generated new TSK 1
I20260925 16:30:46.821246 16372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:30:46.890763 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:46.891175 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:46.891228 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:46.896005 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:46.896114 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:46.896188 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:30:46.902648 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42009
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:46.904223 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:46.905586 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:46.919809 16389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:46.919700 16386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:46.919701 16387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:46.921245 16361 server_base.cc:1034] running on GCE node
I20260925 16:30:46.922035 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:46.923077 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:46.924343 16361 hybrid_clock.cc:648] HybridClock initialized: now 1790353846924303 us; error 72 us; skew 500 ppm
Sep 25 16:30:46 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:46.928836 16361 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:30:46.931077 16361 webserver.cc:460] Webserver started at http://127.0.98.1:42009/ using document root <none> and password file <none>
I20260925 16:30:46.932443 16361 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:46.932643 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:46.939634 16361 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:30:46.943293 16396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:46.945717 16361 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:46.945983 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "afd3d45619e1486a919c25c8a9b4c5de"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "f37afd725c712539b670c8a29ae721a6"
server_key_iv: "d51fc00925c3813bd0b436cb8a3a14b3"
server_key_version: "encryptionkey@0"
I20260925 16:30:46.947187 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:46.972892 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:46.977474 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:46.977684 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:46.978492 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:46.980119 16403 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:30:46.986064 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:30:46.986294 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 16:30:46.986383 16361 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:30:46.988569 16361 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:30:46.988634 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 16:30:46.989164 16403 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap starting.
I20260925 16:30:47.000593 16361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39747
I20260925 16:30:47.000736 16510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39747 every 8 connection(s)
I20260925 16:30:47.001845 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:30:47.011771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16361
I20260925 16:30:47.012305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42909
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33071
--webserver_port=39083
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:30:47.013782 16403 log.cc:824] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:30:47 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:47.025273 16403 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:30:47.025979 16403 tablet_bootstrap.cc:492] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap complete.
I20260925 16:30:47.027112 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.004903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:40105)
0925 16:30:47.005532 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.005593 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.006619 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.008550 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.008559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.008970 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:47.009531 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.009547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.011908 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.011913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.013436 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.013455 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.014712 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.014717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.014928 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.015841 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:47.015865 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:47.018444 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:47.021917 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.021927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.021948 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.022532 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.023241 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.023246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.023248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.023629 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.024573 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:47.024582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:47.025052 (+ 470us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.025324 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.025819 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":167,"threads_started":1}
I20260925 16:30:47.027177 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.004980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40105 (local address 127.0.98.62:33071)
0925 16:30:47.005820 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.005848 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.005934 (+ 86us) server_negotiation.cc:407] Connection header received
0925 16:30:47.007197 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.007230 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.007949 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.008469 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:47.009750 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.011641 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.013633 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.014527 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.014731 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.018879 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:47.018890 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:47.018916 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:47.018970 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:47.021617 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.022717 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.022726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.022733 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.022970 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.023789 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.023793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.023795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.024391 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:47.024542 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.025605 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.026124 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":193,"threads_started":1}
I20260925 16:30:47.027486 16403 ts_tablet_manager.cc:1397] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.014s
I20260925 16:30:47.031020 16511 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33071
I20260925 16:30:47.031303 16511 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:47.030949 16403 raft_consensus.cc:348] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:47.031570 16403 raft_consensus.cc:729] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd3d45619e1486a919c25c8a9b4c5de, State: Initialized, Role: FOLLOWER
I20260925 16:30:47.032037 16511 heartbeater.cc:511] Master 127.0.98.62:33071 requested a full tablet report, sending...
I20260925 16:30:47.032214 16403 consensus_queue.cc:260] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } }
I20260925 16:30:47.033535 16403 ts_tablet_manager.cc:1428] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 16:30:47.036036 16323 ts_manager.cc:194] Registered new tserver with Master: afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747)
I20260925 16:30:47.040650 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40105
I20260925 16:30:47.044077 16511 heartbeater.cc:503] Master 127.0.98.62:33071 was elected leader, sending a full tablet report...
Sep 25 16:30:47 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 16:30:47.055346 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.040215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.98.1:39747)
0925 16:30:47.041227 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.041239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.041264 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:30:47.041343 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.041348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.041555 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.041693 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:47.043037 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.044234 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.045664 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.046254 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.046369 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.049291 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:47.049304 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:47.049320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:47.049348 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:47.051805 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.052783 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.052812 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.052813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.052969 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.053530 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.053533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.053535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.053984 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:47.054132 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.054916 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.055163 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":94,"threads_started":1}
I20260925 16:30:47.055346 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.040098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39747 (local address 127.0.0.1:53780)
0925 16:30:47.040807 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.040864 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.041061 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.041722 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.041728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.042661 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:47.042823 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.042833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.044459 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.044464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.045482 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.045501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.046368 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.046393 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.046538 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.047200 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:47.047213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:47.049094 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:47.052051 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.052071 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.052078 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.052589 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.053094 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.053098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.053100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.053362 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.054136 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:47.054145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:47.054436 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.054748 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.055106 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":178,"threads_started":1}
I20260925 16:30:47.056965 16445 tablet_service.cc:1514] Processing DeleteTablet for tablet fe54dc5053e74468b5a10fa542c3c7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53780
I20260925 16:30:47.058308 16520 tablet_replica.cc:330] stopping tablet replica
I20260925 16:30:47.058725 16520 raft_consensus.cc:2229] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:47.059083 16520 raft_consensus.cc:2258] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:47.060355 16520 ts_tablet_manager.cc:1905] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:47.065315 16520 ts_tablet_manager.cc:1918] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:47.065461 16520 log.cc:1196] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal/wals/fe54dc5053e74468b5a10fa542c3c7b2
I20260925 16:30:47.065707 16520 ts_tablet_manager.cc:1939] T fe54dc5053e74468b5a10fa542c3c7b2 P afd3d45619e1486a919c25c8a9b4c5de: Deleting consensus metadata
I20260925 16:30:47.068161 16310 catalog_manager.cc:4841] TS afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747): tablet fe54dc5053e74468b5a10fa542c3c7b2 (table test-table [id=1887e8959ceb438f93dd3a2309be0d59]) successfully deleted
W20260925 16:30:47.158469 16515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:47.158802 16515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:47.158850 16515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:47.165027 16515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:47.165182 16515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:47.165264 16515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:30:47.172811 16515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39083
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:47.174444 16515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:47.175776 16515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:47.189003 16525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:47.188985 16528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:47.188985 16526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:47.189463 16515 server_base.cc:1034] running on GCE node
I20260925 16:30:47.190074 16515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:47.190973 16515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:47.192198 16515 hybrid_clock.cc:648] HybridClock initialized: now 1790353847192167 us; error 53 us; skew 500 ppm
Sep 25 16:30:47 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:47.196076 16515 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:30:47.197491 16515 webserver.cc:460] Webserver started at http://127.0.98.2:39083/ using document root <none> and password file <none>
I20260925 16:30:47.198341 16515 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:47.198467 16515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:47.204147 16515 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:30:47.207472 16535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:47.209425 16515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:30:47.209640 16515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "20d78789c37b42d3b416ae63867ad677"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "22498727f88bd59adb7b35ab36c41d89"
server_key_iv: "9a27cd9f1b8409c340d70a27b02993d7"
server_key_version: "encryptionkey@0"
I20260925 16:30:47.210529 16515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:47.240798 16515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:47.244962 16515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:47.245229 16515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:47.246062 16515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:47.247159 16515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:47.247210 16515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:47.247268 16515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:47.247293 16515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:47.258760 16515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42909
I20260925 16:30:47.258864 16648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42909 every 8 connection(s)
I20260925 16:30:47.260067 16515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:30:47.261404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16515
I20260925 16:30:47.261910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43495
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33071
--webserver_port=38547
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:30:47 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:47.279174 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.262660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39427 (local address 127.0.98.62:33071)
0925 16:30:47.262843 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.262849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.264638 (+ 1789us) server_negotiation.cc:407] Connection header received
0925 16:30:47.265649 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.265654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.265740 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.265907 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:47.267944 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.268872 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.270813 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.271234 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.271410 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.274759 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:47.274764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:47.274768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:47.274818 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:47.276646 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.277390 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.277394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.277397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.277503 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.277869 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.277872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.277874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.278148 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:47.278250 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.278792 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.278964 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:30:47.279990 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.263473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:39427)
0925 16:30:47.264396 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.264462 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.265386 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.266257 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.266269 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.266918 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:47.267674 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.267698 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.269108 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.269112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.270585 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.270613 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.271519 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.271525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.271768 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.272565 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:47.272585 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:47.274471 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:47.276881 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.276901 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.276926 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.277271 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.277588 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.277590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.277592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.277779 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.278242 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:47.278248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:47.278536 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.278720 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.279035 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":168,"threads_started":1}
I20260925 16:30:47.281459 16649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33071
I20260925 16:30:47.281821 16649 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:47.282428 16649 heartbeater.cc:511] Master 127.0.98.62:33071 requested a full tablet report, sending...
I20260925 16:30:47.284204 16323 ts_manager.cc:194] Registered new tserver with Master: 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909)
I20260925 16:30:47.284955 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39427
W20260925 16:30:47.402449 16651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:30:47.402858 16651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:30:47.402920 16651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:30:47.407613 16651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:30:47.407714 16651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:30:47.407879 16651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:30:47.414701 16651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38547
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:30:47.416229 16651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:30:47.417586 16651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:30:47.429440 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:47.429416 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:30:47.429416 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:30:47.429924 16651 server_base.cc:1034] running on GCE node
I20260925 16:30:47.430441 16651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:30:47.431166 16651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:30:47.432365 16651 hybrid_clock.cc:648] HybridClock initialized: now 1790353847432328 us; error 62 us; skew 500 ppm
Sep 25 16:30:47 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:30:47.435397 16651 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:30:47.436462 16651 webserver.cc:460] Webserver started at http://127.0.98.3:38547/ using document root <none> and password file <none>
I20260925 16:30:47.437103 16651 fs_manager.cc:359] Metadata directory not provided
I20260925 16:30:47.437186 16651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:30:47.441906 16651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:30:47.444713 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:47.446524 16651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:30:47.446688 16651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "d43cb969035c456c93e56891aa471a3a"
format_stamp: "Formatted at 2026-09-25 16:30:26 on dist-test-slave-31kt"
server_key: "350ca0187fffa6fa0d09672c5618947b"
server_key_iv: "ed26c8c5aaeaf59e5e93cafa2d71743f"
server_key_version: "encryptionkey@0"
I20260925 16:30:47.447348 16651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:30:47.468442 16651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:30:47.472088 16651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:30:47.472283 16651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:30:47.472924 16651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:30:47.473927 16651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:30:47.473984 16651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:47.474030 16651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:30:47.474044 16651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:30:47.484264 16651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43495
I20260925 16:30:47.484472 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43495 every 8 connection(s)
I20260925 16:30:47.485628 16651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:30:47.491623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16651
Sep 25 16:30:47 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:30:47.502633 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.492414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.0.98.62:33071)
0925 16:30:47.492769 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.492773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.492938 (+ 165us) server_negotiation.cc:407] Connection header received
0925 16:30:47.493122 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.493125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.493191 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.493287 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:47.494177 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.494662 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.495851 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.496278 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.496422 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.498635 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:47.498640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:47.498643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:47.498701 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:47.500536 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.501052 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.501056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.501058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.501149 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.501418 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.501421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.501423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.501658 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:47.501954 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.502093 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.502242 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":184,"spinlock_wait_cycles":38912,"thread_start_us":65,"threads_started":1}
I20260925 16:30:47.504215 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.488231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41131 (local address 127.0.98.62:33071)
0925 16:30:47.488449 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.488454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.490048 (+ 1594us) server_negotiation.cc:407] Connection header received
0925 16:30:47.491203 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.491206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.491279 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.491424 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:47.493382 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.494015 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.495851 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.496263 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.496426 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.500112 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:47.500118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:47.500121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:47.500160 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:47.501708 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.502313 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.502316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.502318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.502395 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.502939 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.502943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.502944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.503142 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:47.503199 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.503903 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.504030 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":9088}
I20260925 16:30:47.505498 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.488756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:41131)
0925 16:30:47.489726 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.489878 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.490809 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.491733 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.491745 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.492310 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:30:47.493188 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.493215 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.494176 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.494180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.495662 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.495678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.496484 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.496488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.496751 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.497592 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:30:47.497619 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:30:47.499812 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:30:47.501871 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.501879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.501905 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.502214 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.502616 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:30:47.502618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:30:47.502620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:30:47.502822 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:30:47.503270 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:30:47.503276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:30:47.503640 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.503850 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.504306 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":174,"threads_started":1}
I20260925 16:30:47.507087 16781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33071
I20260925 16:30:47.507443 16781 heartbeater.cc:464] Registering TS with master...
I20260925 16:30:47.508136 16781 heartbeater.cc:511] Master 127.0.98.62:33071 requested a full tablet report, sending...
I20260925 16:30:47.509578 16323 ts_manager.cc:194] Registered new tserver with Master: d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
I20260925 16:30:47.510334 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41131
I20260925 16:30:47.517271 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:30:47.522373 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.518057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.0.98.62:33071)
0925 16:30:47.518214 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.518222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.518563 (+ 341us) server_negotiation.cc:407] Connection header received
0925 16:30:47.518764 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.518768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.518895 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.519012 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:30:47.519305 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.519923 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.520927 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.521205 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.521326 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.521735 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:30:47.521959 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 16:30:47.522240 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35808: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 16:30:47.522461 16514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35808: token expired
W20260925 16:30:47.523036 16251 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33071)
I20260925 16:30:47.532826 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.523548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.0.98.62:33071)
0925 16:30:47.523693 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.523698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.523766 (+ 68us) server_negotiation.cc:407] Connection header received
0925 16:30:47.523915 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.523918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.523965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.524029 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:30:47.524746 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.525522 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.526483 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.526711 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.526799 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.528370 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:30:47.528373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:30:47.528376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:30:47.528408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:30:47.530846 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.531548 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.531551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.531553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.531647 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:30:47.531932 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:30:47.531935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:30:47.531937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:30:47.532146 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:30:47.532279 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.532520 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.532662 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:30:47.536350 16323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:30:47.540208 16323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:30:47.551147 16445 tablet_service.cc:1467] Processing CreateTablet for tablet e35d8f83de2140eba36c7ece91bc5208 (DEFAULT_TABLE table=test-table [id=68b1628e9fdc454ca56584bb55b7befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:47.552112 16445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e35d8f83de2140eba36c7ece91bc5208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:47.556097 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.549810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42909 (local address 127.0.0.1:38034)
0925 16:30:47.549968 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.550017 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.550139 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.551426 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.551429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.551454 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:47.551643 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.551652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.553358 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.553362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.554517 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.554529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.555593 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.555597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.555705 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.555774 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.555830 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.555926 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":65}
I20260925 16:30:47.556627 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.550067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43495 (local address 127.0.0.1:47048)
0925 16:30:47.550488 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.550504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.550625 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.551663 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.551666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.551690 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:47.551836 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.551843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.553425 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.553429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.554804 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.554818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.556108 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.556112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.556225 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.556259 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.556312 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.556418 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":68,"thread_start_us":143,"threads_started":1}
I20260925 16:30:47.557031 16793 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap starting.
I20260925 16:30:47.557399 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.550228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.98.2:42909)
0925 16:30:47.550821 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.550834 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.550860 (+ 26us) server_negotiation.cc:407] Connection header received
0925 16:30:47.550952 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.550958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.551229 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.551430 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:47.551833 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.553174 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.554679 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.555459 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.556644 (+ 1185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.557024 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.557101 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.557246 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":152,"threads_started":1}
I20260925 16:30:47.557402 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.550336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.0.98.3:43495)
0925 16:30:47.550939 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.550951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.550971 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:30:47.551041 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.551045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.551217 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.551415 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:47.551975 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.553256 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.554995 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.555931 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.556862 (+ 931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.557040 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.557103 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.557239 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":269,"threads_started":1}
I20260925 16:30:47.558414 16793 tablet_bootstrap.cc:654] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:47.560207 16793 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: No bootstrap required, opened a new log
I20260925 16:30:47.560345 16793 ts_tablet_manager.cc:1397] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 16:30:47.560937 16793 raft_consensus.cc:348] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.561111 16793 raft_consensus.cc:374] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:47.561154 16793 raft_consensus.cc:729] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd3d45619e1486a919c25c8a9b4c5de, State: Initialized, Role: FOLLOWER
I20260925 16:30:47.561372 16793 consensus_queue.cc:260] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.561444 16583 tablet_service.cc:1467] Processing CreateTablet for tablet e35d8f83de2140eba36c7ece91bc5208 (DEFAULT_TABLE table=test-table [id=68b1628e9fdc454ca56584bb55b7befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:47.561517 16715 tablet_service.cc:1467] Processing CreateTablet for tablet e35d8f83de2140eba36c7ece91bc5208 (DEFAULT_TABLE table=test-table [id=68b1628e9fdc454ca56584bb55b7befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:30:47.562476 16793 ts_tablet_manager.cc:1428] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 16:30:47.562803 16583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e35d8f83de2140eba36c7ece91bc5208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:47.562959 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e35d8f83de2140eba36c7ece91bc5208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:30:47.571130 16796 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Bootstrap starting.
I20260925 16:30:47.572190 16797 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Bootstrap starting.
I20260925 16:30:47.574569 16796 tablet_bootstrap.cc:654] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:47.574637 16797 tablet_bootstrap.cc:654] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Neither blocks nor log segments found. Creating new log.
I20260925 16:30:47.575700 16796 log.cc:824] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:47.575783 16797 log.cc:824] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:30:47.578523 16796 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: No bootstrap required, opened a new log
I20260925 16:30:47.578639 16797 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: No bootstrap required, opened a new log
I20260925 16:30:47.578771 16796 ts_tablet_manager.cc:1397] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 16:30:47.578910 16797 ts_tablet_manager.cc:1397] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:30:47.582384 16797 raft_consensus.cc:348] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.582386 16796 raft_consensus.cc:348] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.582759 16796 raft_consensus.cc:374] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:47.582762 16797 raft_consensus.cc:374] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:30:47.582818 16796 raft_consensus.cc:729] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d78789c37b42d3b416ae63867ad677, State: Initialized, Role: FOLLOWER
I20260925 16:30:47.582818 16797 raft_consensus.cc:729] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43cb969035c456c93e56891aa471a3a, State: Initialized, Role: FOLLOWER
I20260925 16:30:47.583443 16796 consensus_queue.cc:260] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.583443 16797 consensus_queue.cc:260] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.584465 16781 heartbeater.cc:503] Master 127.0.98.62:33071 was elected leader, sending a full tablet report...
I20260925 16:30:47.585075 16797 ts_tablet_manager.cc:1428] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 16:30:47.585732 16796 ts_tablet_manager.cc:1428] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:30:47.586191 16649 heartbeater.cc:503] Master 127.0.98.62:33071 was elected leader, sending a full tablet report...
I20260925 16:30:47.710139 16801 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:30:47.710466 16801 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.712374 16801 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747)
I20260925 16:30:47.718083 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.712677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42909 (local address 127.0.98.3:38643)
0925 16:30:47.713186 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.713199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.713300 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.713640 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.713642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.713689 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:47.713916 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.713924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.715103 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.715108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.716442 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.716457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.717495 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.717499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.717605 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.717639 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.717731 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.717871 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":287,"spinlock_wait_cycles":43008,"thread_start_us":118,"threads_started":1}
I20260925 16:30:47.718061 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.712831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38643 (local address 127.0.98.2:42909)
0925 16:30:47.712998 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.713003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.713323 (+ 320us) server_negotiation.cc:407] Connection header received
0925 16:30:47.713401 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.713404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.713458 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.713561 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:47.714072 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.714895 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.716622 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.717316 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.717469 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.717562 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.717779 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.717922 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 16:30:47.718340 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.712846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52827 (local address 127.0.98.1:39747)
0925 16:30:47.713187 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.713191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.713211 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:30:47.713339 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.713342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.713418 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.713561 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:30:47.714072 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.714961 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.716670 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.717533 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.717660 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.717901 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.717942 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.718214 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260925 16:30:47.718338 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.712673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39747 (local address 127.0.98.3:52827)
0925 16:30:47.713014 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.713032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:30:47.713186 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:30:47.713635 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:30:47.713639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:30:47.713691 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:30:47.713916 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.713925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.715149 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.715153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.716441 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:30:47.716456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.717677 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.717681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:30:47.717827 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.717847 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:30:47.717912 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 16:30:47.718145 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":148,"spinlock_wait_cycles":6656}
I20260925 16:30:47.719416 16603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35d8f83de2140eba36c7ece91bc5208" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d78789c37b42d3b416ae63867ad677" is_pre_election: true
I20260925 16:30:47.719393 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35d8f83de2140eba36c7ece91bc5208" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd3d45619e1486a919c25c8a9b4c5de" is_pre_election: true
I20260925 16:30:47.719830 16603 raft_consensus.cc:2454] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 0.
I20260925 16:30:47.719830 16465 raft_consensus.cc:2454] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 0.
I20260925 16:30:47.720397 16671 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de, d43cb969035c456c93e56891aa471a3a; no voters:
I20260925 16:30:47.720777 16801 raft_consensus.cc:2781] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:30:47.720929 16801 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:30:47.720964 16801 raft_consensus.cc:3037] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:47.722321 16801 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.722810 16801 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 election: Requested vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747)
I20260925 16:30:47.723482 16603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35d8f83de2140eba36c7ece91bc5208" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d78789c37b42d3b416ae63867ad677"
I20260925 16:30:47.723475 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35d8f83de2140eba36c7ece91bc5208" candidate_uuid: "d43cb969035c456c93e56891aa471a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd3d45619e1486a919c25c8a9b4c5de"
I20260925 16:30:47.723680 16603 raft_consensus.cc:3037] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:47.723680 16465 raft_consensus.cc:3037] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:30:47.725346 16603 raft_consensus.cc:2454] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 1.
I20260925 16:30:47.725337 16465 raft_consensus.cc:2454] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43cb969035c456c93e56891aa471a3a in term 1.
I20260925 16:30:47.725817 16671 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de, d43cb969035c456c93e56891aa471a3a; no voters:
I20260925 16:30:47.726053 16801 raft_consensus.cc:2781] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:30:47.726351 16801 raft_consensus.cc:686] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 1 LEADER]: Becoming Leader. State: Replica: d43cb969035c456c93e56891aa471a3a, State: Running, Role: LEADER
I20260925 16:30:47.726787 16801 consensus_queue.cc:237] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:30:47.731536 16323 catalog_manager.cc:5495] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to d43cb969035c456c93e56891aa471a3a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d43cb969035c456c93e56891aa471a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } health_report { overall_health: UNKNOWN } } }
W20260925 16:30:47.736914 16782 tablet.cc:2369] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:30:47.753242 16512 tablet.cc:2369] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:47.759552 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.755116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.0.98.3:43495)
0925 16:30:47.755254 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:30:47.755258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:30:47.755277 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:30:47.755454 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:30:47.755456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:30:47.755523 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:30:47.755626 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:30:47.756038 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.756648 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.757685 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:30:47.758021 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:30:47.758141 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:30:47.758561 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:30:47.759056 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:30:47.759178 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:30:47.759307 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 16:30:47.759379 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 16:30:47.761514 16650 tablet.cc:2369] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:30:47.767901 16603 raft_consensus.cc:1261] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Refusing update from remote peer d43cb969035c456c93e56891aa471a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:47.768095 16465 raft_consensus.cc:1261] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Refusing update from remote peer d43cb969035c456c93e56891aa471a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:30:47.769203 16801 consensus_queue.cc:1035] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:47.769538 16804 consensus_queue.cc:1035] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:30:47.778564 16807 mvcc.cc:205] Tried to move back new op lower bound from 7333289360445112320 to 7333289360292683776. Current Snapshot: MvccSnapshot[applied={T|T < 7333289360445112320}]
I20260925 16:30:47.792261 16323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:47.792510 16323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:30:47.796322 16323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Sending DeleteTablet for 3 replicas of tablet e35d8f83de2140eba36c7ece91bc5208
I20260925 16:30:47.797340 16445 tablet_service.cc:1514] Processing DeleteTablet for tablet e35d8f83de2140eba36c7ece91bc5208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53780
I20260925 16:30:47.797482 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet e35d8f83de2140eba36c7ece91bc5208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47048
I20260925 16:30:47.797570 16583 tablet_service.cc:1514] Processing DeleteTablet for tablet e35d8f83de2140eba36c7ece91bc5208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:30:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38034
I20260925 16:30:47.798229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 16361
I20260925 16:30:47.798226 16817 tablet_replica.cc:330] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: stopping tablet replica
I20260925 16:30:47.799023 16818 tablet_replica.cc:330] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: stopping tablet replica
I20260925 16:30:47.799654 16818 raft_consensus.cc:2229] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:30:47.799979 16817 raft_consensus.cc:2229] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:30:47.800328 16818 raft_consensus.cc:2258] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:47.800474 16817 raft_consensus.cc:2258] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:30:47.802436 16817 ts_tablet_manager.cc:1905] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:47.802692 16818 ts_tablet_manager.cc:1905] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:30:47.806134 16817 ts_tablet_manager.cc:1918] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:47.806267 16817 log.cc:1196] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/e35d8f83de2140eba36c7ece91bc5208
I20260925 16:30:47.806665 16817 ts_tablet_manager.cc:1939] T e35d8f83de2140eba36c7ece91bc5208 P d43cb969035c456c93e56891aa471a3a: Deleting consensus metadata
I20260925 16:30:47.806686 16818 ts_tablet_manager.cc:1918] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:30:47.807035 16818 log.cc:1196] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/e35d8f83de2140eba36c7ece91bc5208
I20260925 16:30:47.807427 16818 ts_tablet_manager.cc:1939] T e35d8f83de2140eba36c7ece91bc5208 P 20d78789c37b42d3b416ae63867ad677: Deleting consensus metadata
I20260925 16:30:47.808569 16307 catalog_manager.cc:4841] TS d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): tablet e35d8f83de2140eba36c7ece91bc5208 (table test-table [id=68b1628e9fdc454ca56584bb55b7befb]) successfully deleted
I20260925 16:30:47.809273 16307 catalog_manager.cc:4841] TS 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): tablet e35d8f83de2140eba36c7ece91bc5208 (table test-table [id=68b1628e9fdc454ca56584bb55b7befb]) successfully deleted
W20260925 16:30:47.809392 16310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39747 (error 108)
I20260925 16:30:47.810667 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:30:47.810345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39747 (local address 127.0.0.1:53786)
0925 16:30:47.810486 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 16:30:47.810565 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 16:30:47.810909 16310 catalog_manager.cc:4584] TS afd3d45619e1486a919c25c8a9b4c5de (127.0.98.1:39747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e35d8f83de2140eba36c7ece91bc5208: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39747: connect: Connection refused (error 111)
I20260925 16:30:47.811051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 16515
I20260925 16:30:47.818769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 16651
I20260925 16:30:47.826238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 16290
I20260925 16:31:07.834743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--webserver_interface=127.0.98.62
--webserver_port=44333
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:07.971217 16820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:07.971880 16820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:07.972019 16820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:07.977548 16820 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 16:31:07.977661 16820 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 16:31:07.977679 16820 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 16:31:07.977691 16820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:07.977710 16820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:07.977730 16820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:07.977746 16820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:07.977798 16820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:07.983996 16820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:07.985265 16820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:07.986481 16820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:07.997115 16828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:07.997077 16825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:07.997115 16826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:07.997294 16820 server_base.cc:1034] running on GCE node
I20260925 16:31:07.998382 16820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:07.999672 16820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:08.000905 16820 hybrid_clock.cc:648] HybridClock initialized: now 1790353868000870 us; error 63 us; skew 500 ppm
Sep 25 16:31:08 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:08.011025 16820 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:08.012223 16820 webserver.cc:460] Webserver started at http://127.0.98.62:44333/ using document root <none> and password file <none>
I20260925 16:31:08.012832 16820 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:08.012907 16820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:08.017657 16820 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 16:31:08.020608 16835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.022753 16820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:08.022954 16820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "c9194492681e48788a4c42a52436f501"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "c9fc6a926c51542f0f7f70e337802d59"
server_key_iv: "527ca60182c06b7217e16d374bff1752"
server_key_version: "encryptionkey@0"
I20260925 16:31:08.023667 16820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:08.050575 16820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:08.055680 16820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:08.055958 16820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:08.066294 16820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33071
I20260925 16:31:08.066578 16887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33071 every 8 connection(s)
I20260925 16:31:08.067953 16820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:08.074766 16888 sys_catalog.cc:263] Verifying existing consensus state
I20260925 16:31:08.074926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16820
I20260925 16:31:08.075583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39747
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33071
--webserver_port=42009
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:31:08.077574 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap starting.
I20260925 16:31:08.097420 16888 log.cc:824] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:08.109920 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:31:08.110383 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Bootstrap complete.
I20260925 16:31:08.114989 16888 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:31:08.115474 16888 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9194492681e48788a4c42a52436f501, State: Initialized, Role: FOLLOWER
I20260925 16:31:08.116055 16888 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:31:08.116199 16888 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:08.116271 16888 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:08.116389 16888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 2 FOLLOWER]: Advancing to term 3
I20260925 16:31:08.118970 16888 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:31:08.119426 16888 leader_election.cc:302] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9194492681e48788a4c42a52436f501; no voters:
I20260925 16:31:08.119864 16888 leader_election.cc:288] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 16:31:08.119985 16892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 16:31:08.120194 16892 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [term 3 LEADER]: Becoming Leader. State: Replica: c9194492681e48788a4c42a52436f501, State: Running, Role: LEADER
I20260925 16:31:08.120559 16892 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260925 16:31:08.121330 16888 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:08.121685 16892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9194492681e48788a4c42a52436f501. Latest consensus state: current_term: 3 leader_uuid: "c9194492681e48788a4c42a52436f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260925 16:31:08.121851 16892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:08.122529 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9194492681e48788a4c42a52436f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9194492681e48788a4c42a52436f501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260925 16:31:08.122646 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:08.123628 16896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:08.131038 16896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:08.131982 16896 catalog_manager.cc:1259] Loaded cluster ID: f4531ea0080e407fb172eaca91a58f44
I20260925 16:31:08.132061 16896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:08.135022 16896 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:08.135944 16896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Loaded TSK: 0
I20260925 16:31:08.136040 16896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9194492681e48788a4c42a52436f501: Loaded TSK: 1
F20260925 16:31:08.136893 16896 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f78a11bf9ad google::LogMessage::Fail() at ??:0
@ 0x7f78a11c288a google::LogMessage::SendToLog() at ??:0
@ 0x7f78a11bf4de google::LogMessage::Flush() at ??:0
@ 0x7f78a11c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f78a3e8318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f78a3e7bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f78abb247bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f78abafe99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f78abaff178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f78abb40df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f78abb5f82b std::function<>::operator()() at ??:0
@ 0x7f78abafeb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f78abafff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f78abafc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f78abb40640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f78a34113d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f78a3411c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f78a34133f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f78a3402512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f78a4a466db start_thread at ??:0
@ 0x7f789fcbd61f clone at ??:0
W20260925 16:31:08.260080 16890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:08.260627 16890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:08.260708 16890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:08.268075 16890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:08.268219 16890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:08.268323 16890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:08.278585 16890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42009
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:08.280586 16890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:08.282523 16890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:08.296759 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:08.296905 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:08.296765 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:08.297365 16890 server_base.cc:1034] running on GCE node
I20260925 16:31:08.297950 16890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:08.298763 16890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:08.300068 16890 hybrid_clock.cc:648] HybridClock initialized: now 1790353868300035 us; error 62 us; skew 500 ppm
Sep 25 16:31:08 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:08.303519 16890 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:31:08.304663 16890 webserver.cc:460] Webserver started at http://127.0.98.1:42009/ using document root <none> and password file <none>
I20260925 16:31:08.305291 16890 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:08.305361 16890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:08.309697 16890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:31:08.312453 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.314538 16890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:08.314730 16890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "afd3d45619e1486a919c25c8a9b4c5de"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "f37afd725c712539b670c8a29ae721a6"
server_key_iv: "d51fc00925c3813bd0b436cb8a3a14b3"
server_key_version: "encryptionkey@0"
I20260925 16:31:08.315392 16890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:08.327067 16890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:08.331471 16890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:08.331727 16890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:08.332549 16890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:08.334038 16952 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 16:31:08.340073 16890 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 16:31:08.340193 16890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 16:31:08.340267 16890 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 16:31:08.342758 16890 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 16:31:08.342859 16890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260925 16:31:08.343173 16952 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap starting.
I20260925 16:31:08.355845 16890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39747
I20260925 16:31:08.355937 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39747 every 8 connection(s)
I20260925 16:31:08.357415 16890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:08.365684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 16890
I20260925 16:31:08.366286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42909
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33071
--webserver_port=39083
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:31:08.367003 16952 log.cc:824] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:08.376572 16952 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 16:31:08.377059 16952 tablet_bootstrap.cc:492] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Bootstrap complete.
I20260925 16:31:08.378553 16952 ts_tablet_manager.cc:1397] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.012s
I20260925 16:31:08.382411 16952 raft_consensus.cc:348] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:08.383064 16952 raft_consensus.cc:729] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd3d45619e1486a919c25c8a9b4c5de, State: Initialized, Role: FOLLOWER
I20260925 16:31:08.383723 16952 consensus_queue.cc:260] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:08.384994 16952 ts_tablet_manager.cc:1428] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260925 16:31:08.419298 17062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:31:08.360846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:60493)
0925 16:31:08.361891 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.361978 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:08.363386 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:08.417924 (+ 54538us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":400,"threads_started":1}
W20260925 16:31:08.420051 17060 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33071 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33071: Client connection negotiation failed: client connection to 127.0.98.62:33071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 16:31:08.421216 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.420830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:53713)
0925 16:31:08.421009 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.421103 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 16:31:08.422407 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.422056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:46533)
0925 16:31:08.422217 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.422303 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 16:31:08.422641 17060 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:31:08.510430 17064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:08.510749 17064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:08.510798 17064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:08.515024 17064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:08.515110 17064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:08.515173 17064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:08.521107 17064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39083
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:08.522307 17064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:08.523345 17064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:08.534240 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:08.534348 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:08.534363 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:08.534554 17064 server_base.cc:1034] running on GCE node
I20260925 16:31:08.535054 17064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:08.535789 17064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:08.537014 17064 hybrid_clock.cc:648] HybridClock initialized: now 1790353868536981 us; error 51 us; skew 500 ppm
Sep 25 16:31:08 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:08.540203 17064 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:31:08.541286 17064 webserver.cc:460] Webserver started at http://127.0.98.2:39083/ using document root <none> and password file <none>
I20260925 16:31:08.541961 17064 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:08.542040 17064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:08.546597 17064 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:31:08.549221 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.551189 17064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:31:08.551376 17064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "20d78789c37b42d3b416ae63867ad677"
format_stamp: "Formatted at 2026-09-25 16:30:25 on dist-test-slave-31kt"
server_key: "22498727f88bd59adb7b35ab36c41d89"
server_key_iv: "9a27cd9f1b8409c340d70a27b02993d7"
server_key_version: "encryptionkey@0"
I20260925 16:31:08.552049 17064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:08.566623 17064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:08.570320 17064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:08.570518 17064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:08.571195 17064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:08.572219 17064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:08.572274 17064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.572320 17064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:08.572352 17064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.582808 17064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42909
I20260925 16:31:08.582855 17193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42909 every 8 connection(s)
I20260925 16:31:08.584124 17064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:08.589635 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.587191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:49943)
0925 16:31:08.587913 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.588094 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":134,"threads_started":1}
W20260925 16:31:08.590476 17194 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33071 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33071: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
I20260925 16:31:08.590576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17064
I20260925 16:31:08.591087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43495
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33071
--webserver_port=38547
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:31:08.591483 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.591097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:37137)
0925 16:31:08.591301 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.591385 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 16:31:08.592238 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.592010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:47125)
0925 16:31:08.592129 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.592177 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 16:31:08.592442 17194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 16:31:08.726788 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:08.727167 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:08.727218 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:08.731895 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:08.731990 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:08.732062 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:08.738400 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38547
--tserver_master_addrs=127.0.98.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:08.739979 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:08.741310 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:08.753809 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:08.754104 17197 server_base.cc:1034] running on GCE node
W20260925 16:31:08.753887 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:08.753809 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:08.754856 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:08.755681 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:08.756896 17197 hybrid_clock.cc:648] HybridClock initialized: now 1790353868756875 us; error 41 us; skew 500 ppm
Sep 25 16:31:08 dist-test-slave-31kt krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:08.760152 17197 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:31:08.761237 17197 webserver.cc:460] Webserver started at http://127.0.98.3:38547/ using document root <none> and password file <none>
I20260925 16:31:08.761921 17197 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:08.761997 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:08.766850 17197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:31:08.769708 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.771534 17197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 16:31:08.771723 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "d43cb969035c456c93e56891aa471a3a"
format_stamp: "Formatted at 2026-09-25 16:30:26 on dist-test-slave-31kt"
server_key: "350ca0187fffa6fa0d09672c5618947b"
server_key_iv: "ed26c8c5aaeaf59e5e93cafa2d71743f"
server_key_version: "encryptionkey@0"
I20260925 16:31:08.772588 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:08.786358 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:08.790091 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:08.790272 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:08.790900 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:08.791864 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:08.791920 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.791961 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:08.791976 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:08.802321 17197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43495
I20260925 16:31:08.802385 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43495 every 8 connection(s)
I20260925 16:31:08.803763 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:08.809439 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.807083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:57339)
0925 16:31:08.808182 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.808351 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":222,"threads_started":1}
W20260925 16:31:08.810403 17326 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33071 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33071: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
I20260925 16:31:08.811154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17197
I20260925 16:31:08.811865 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.811393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:40413)
0925 16:31:08.811652 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.811761 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 16:31:08.812870 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:08.812543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:42635)
0925 16:31:08.812709 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:08.812784 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 16:31:08.813210 17326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 16:31:09.424194 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:09.423535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:46645)
0925 16:31:09.423923 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:09.424025 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":92,"threads_started":1}
I20260925 16:31:09.594112 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:09.593369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:59305)
0925 16:31:09.593850 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:09.593963 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":104,"threads_started":1}
I20260925 16:31:09.814839 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:09.814201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:48103)
0925 16:31:09.814608 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:09.814699 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":107,"threads_started":1}
I20260925 16:31:10.102785 17333 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:10.103077 17333 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:10.104499 17333 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
Sep 25 16:31:10 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 16:31:10 dist-test-slave-31kt krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 16:31:10.122328 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.105301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43495 (local address 127.0.98.1:54925)
0925 16:31:10.105690 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:10.105704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:10.105855 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:10.108473 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:10.108476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:10.108703 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:10.109288 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:10.109309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.110875 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.110878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:10.112211 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:10.112231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.113582 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.113585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:10.113660 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:10.114359 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:10.114372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:10.116322 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:10.119298 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:10.119307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:10.119327 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:10.119749 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:10.120187 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:10.120191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:10.120193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:10.120363 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:10.121132 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:10.121135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:10.121456 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 16:31:10.121667 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 16:31:10.122098 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":236,"spinlock_wait_cycles":48128,"thread_start_us":77,"threads_started":1}
I20260925 16:31:10.122347 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.105582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54925 (local address 127.0.98.3:43495)
0925 16:31:10.105970 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:10.105999 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:10.106037 (+ 38us) server_negotiation.cc:407] Connection header received
0925 16:31:10.106845 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:10.106854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:10.107455 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:10.107881 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:10.109482 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.110722 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.112372 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.113139 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.113327 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:10.116667 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:10.116674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:10.116692 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:10.116735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:10.119122 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:10.119917 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:10.119925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:10.119931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:10.120088 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:10.120457 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:10.120461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:10.120463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:10.121020 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:10.121156 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:10.121833 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 16:31:10.122152 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260925 16:31:10.122927 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.105106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35163 (local address 127.0.98.2:42909)
0925 16:31:10.105656 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:10.105673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:10.105727 (+ 54us) server_negotiation.cc:407] Connection header received
0925 16:31:10.106845 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:10.106854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:10.107456 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:10.107881 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:10.109581 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.110737 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.112341 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.113140 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.113327 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:10.116962 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:10.116970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:10.116989 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:10.117032 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:10.119126 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:10.119941 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:10.119947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:10.119951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:10.120091 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:10.120470 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:10.120473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:10.120475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:10.120987 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:10.121112 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:10.122325 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0925 16:31:10.122718 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":108,"threads_started":1}
I20260925 16:31:10.123471 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.104858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42909 (local address 127.0.98.1:35163)
0925 16:31:10.105220 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:10.105246 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:10.105415 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:10.108084 (+ 2669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:10.108092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:10.108645 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:10.109289 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:10.109312 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.110939 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.110942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:10.112189 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:10.112202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:10.113311 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:10.113314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:10.113503 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:10.114250 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:10.114277 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:10.116657 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:10.119305 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:10.119309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:10.119330 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:10.119796 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:10.120200 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:10.120204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:10.120206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:10.120380 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:10.121125 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:10.121133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:10.121458 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 16:31:10.122035 (+ 577us) client_negotiation.cc:240] Negotiation successful
0925 16:31:10.123288 (+ 1253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":181,"spinlock_wait_cycles":8192,"thread_start_us":118,"threads_started":1}
W20260925 16:31:10.125537 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
W20260925 16:31:10.126093 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
I20260925 16:31:10.126200 16946 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de; no voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a
I20260925 16:31:10.126667 17333 raft_consensus.cc:2726] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:31:10.425807 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.425407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:36749)
0925 16:31:10.425610 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:10.425669 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 16:31:10.596123 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.595384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:48779)
0925 16:31:10.595834 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:10.595937 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":99,"threads_started":1}
I20260925 16:31:10.816560 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:10.815959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:44845)
0925 16:31:10.816359 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:10.816446 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260925 16:31:11.427608 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:11.426939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:59745)
0925 16:31:11.427402 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:11.427491 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":130,"threads_started":1}
I20260925 16:31:11.598155 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:11.597412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:44497)
0925 16:31:11.597859 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:11.597962 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":94,"threads_started":1}
I20260925 16:31:11.818346 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:11.817745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:53569)
0925 16:31:11.818138 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:11.818222 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":97,"threads_started":1}
I20260925 16:31:12.063555 17343 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:12.063716 17343 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:12.064211 17343 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
W20260925 16:31:12.065404 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
W20260925 16:31:12.065594 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
I20260925 16:31:12.065650 16946 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de; no voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a
I20260925 16:31:12.065939 17343 raft_consensus.cc:2726] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:31:12.429500 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:12.428872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:54287)
0925 16:31:12.429250 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:12.429368 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260925 16:31:12.599829 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:12.599253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:55391)
0925 16:31:12.599608 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:12.599695 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
I20260925 16:31:12.819907 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:12.819233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:48717)
0925 16:31:12.819694 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:12.819762 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":111,"threads_started":1}
I20260925 16:31:13.431411 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:13.430753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:42693)
0925 16:31:13.431179 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:13.431267 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":89,"threads_started":1}
I20260925 16:31:13.601910 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:13.601179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:36213)
0925 16:31:13.601613 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:13.601710 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":108,"threads_started":1}
I20260925 16:31:13.821509 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:13.820966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:52933)
0925 16:31:13.821330 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:13.821407 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260925 16:31:14.433037 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:14.432563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:36473)
0925 16:31:14.432858 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:14.432925 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
I20260925 16:31:14.603545 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:14.602972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:40573)
0925 16:31:14.603351 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:14.603410 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":201,"threads_started":1}
I20260925 16:31:14.823071 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:14.822436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:39795)
0925 16:31:14.822837 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:14.822934 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260925 16:31:14.972218 17353 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:14.972419 17353 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:14.972994 17353 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
W20260925 16:31:14.974100 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
W20260925 16:31:14.974298 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
I20260925 16:31:14.974359 16946 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de; no voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a
I20260925 16:31:14.974536 17353 raft_consensus.cc:2726] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:31:15.434616 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:15.434073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:59985)
0925 16:31:15.434408 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:15.434481 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260925 16:31:15.605402 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:15.604694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:40559)
0925 16:31:15.605168 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:15.605263 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":106,"threads_started":1}
I20260925 16:31:15.824927 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:15.824237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:55653)
0925 16:31:15.824717 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:15.824814 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":117,"threads_started":1}
I20260925 16:31:16.436182 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:16.435556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:38051)
0925 16:31:16.435952 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:16.436044 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":88,"threads_started":1}
I20260925 16:31:16.607095 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:16.606508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:42469)
0925 16:31:16.606866 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:16.606956 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260925 16:31:16.826748 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:16.826117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:41001)
0925 16:31:16.826510 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:16.826610 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":95,"threads_started":1}
I20260925 16:31:17.438189 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:17.437466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:43183)
0925 16:31:17.437907 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:17.438036 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":142,"threads_started":1}
I20260925 16:31:17.592190 17361 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:17.592357 17361 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:17.592867 17361 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
W20260925 16:31:17.593806 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
W20260925 16:31:17.593953 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
I20260925 16:31:17.594002 16946 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de; no voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a
I20260925 16:31:17.594149 17361 raft_consensus.cc:2726] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:31:17.608644 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:17.608038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:51229)
0925 16:31:17.608424 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:17.608506 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":176,"threads_started":1}
I20260925 16:31:17.828390 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:17.827793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:47067)
0925 16:31:17.828141 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:17.828261 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":89,"threads_started":1}
I20260925 16:31:18.440026 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:18.439301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:49419)
0925 16:31:18.439805 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:18.439897 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":217,"threads_started":1}
I20260925 16:31:18.610646 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:18.610003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:60661)
0925 16:31:18.610409 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:18.610511 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260925 16:31:18.829989 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:18.829369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:33543)
0925 16:31:18.829749 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:18.829862 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":176,"threads_started":1}
I20260925 16:31:19.441877 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:19.441279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:45077)
0925 16:31:19.441621 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:19.441695 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260925 16:31:19.612528 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:19.611900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:55427)
0925 16:31:19.612294 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:19.612391 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":109,"threads_started":1}
I20260925 16:31:19.831624 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:19.831036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:53077)
0925 16:31:19.831389 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:19.831474 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":89,"threads_started":1}
I20260925 16:31:20.443538 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:20.442956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:34055)
0925 16:31:20.443326 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:20.443396 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260925 16:31:20.614377 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:20.613649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:55229)
0925 16:31:20.614089 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:20.614203 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":133,"threads_started":1}
I20260925 16:31:20.833359 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:20.832756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:56837)
0925 16:31:20.833144 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:20.833236 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260925 16:31:21.445350 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:21.444713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:43447)
0925 16:31:21.445122 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:21.445214 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260925 16:31:21.616212 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:21.615617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:47085)
0925 16:31:21.616011 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:21.616083 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":161,"threads_started":1}
I20260925 16:31:21.835052 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:21.834404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:60725)
0925 16:31:21.834825 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:21.834913 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20260925 16:31:22.447088 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:22.446521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:54165)
0925 16:31:22.446887 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:22.446970 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260925 16:31:22.617697 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:22.617133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:49947)
0925 16:31:22.617470 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:22.617567 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260925 16:31:22.836954 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:22.836292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.3:45137)
0925 16:31:22.836679 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:22.836786 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260925 16:31:23.449007 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:23.448381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.1:53729)
0925 16:31:23.448799 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:23.448888 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":116,"threads_started":1}
I20260925 16:31:23.619601 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:23.619009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33071 (local address 127.0.98.2:50969)
0925 16:31:23.619349 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:23.619428 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260925 16:31:23.786799 17381 raft_consensus.cc:482] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:23.786998 17381 raft_consensus.cc:504] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d78789c37b42d3b416ae63867ad677" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42909 } } peers { permanent_uuid: "d43cb969035c456c93e56891aa471a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43495 } } peers { permanent_uuid: "afd3d45619e1486a919c25c8a9b4c5de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39747 } }
I20260925 16:31:23.787616 17381 leader_election.cc:288] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909), d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495)
W20260925 16:31:23.788662 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d78789c37b42d3b416ae63867ad677 (127.0.98.2:42909): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
W20260925 16:31:23.789155 16946 leader_election.cc:341] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d43cb969035c456c93e56891aa471a3a (127.0.98.3:43495): Not found: Tablet not found: e35d8f83de2140eba36c7ece91bc5208
I20260925 16:31:23.789281 16946 leader_election.cc:302] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afd3d45619e1486a919c25c8a9b4c5de; no voters: 20d78789c37b42d3b416ae63867ad677, d43cb969035c456c93e56891aa471a3a
I20260925 16:31:23.789485 17381 raft_consensus.cc:2726] T e35d8f83de2140eba36c7ece91bc5208 P afd3d45619e1486a919c25c8a9b4c5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 16:31:23.822821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 16890
I20260925 16:31:23.829690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17064
I20260925 16:31:23.836726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17197
2026-09-25T16:31:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59870 ms)
[----------] 30 tests from SecurityITest (271393 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:23 dist-test-slave-31kt krb5kdc[17384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:23 dist-test-slave-31kt krb5kdc[17384](info): set up 2 sockets
Sep 25 16:31:23 dist-test-slave-31kt krb5kdc[17384](info): commencing operation
krb5kdc: starting...
W20260925 16:31:24.830222 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:24 dist-test-slave-31kt krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:24.970085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:25.107147 17397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:25.107563 17397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:25.107627 17397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:25.112448 17397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:25.112558 17397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:25.112588 17397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:25.112612 17397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:25.112641 17397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:25.119009 17397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:25.120497 17397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:25.121842 17397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:25.132829 17402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:25.132977 17397 server_base.cc:1034] running on GCE node
W20260925 16:31:25.132844 17405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:25.132829 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:25.134016 17397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:25.135517 17397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:25.138890 17397 hybrid_clock.cc:648] HybridClock initialized: now 1790353885138834 us; error 62 us; skew 500 ppm
Sep 25 16:31:25 dist-test-slave-31kt krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:25.143169 17397 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:25.144755 17397 webserver.cc:460] Webserver started at http://127.0.98.62:43769/ using document root <none> and password file <none>
I20260925 16:31:25.145619 17397 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:25.145720 17397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:25.146045 17397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:25.148941 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "8650ef1ba2d24234981a633a633b260a"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "13a7bc08480852a6fef9e6dbec8ac468"
server_key_iv: "2ef4969c2f9db610584640ca9c0f526f"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.149854 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "8650ef1ba2d24234981a633a633b260a"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "13a7bc08480852a6fef9e6dbec8ac468"
server_key_iv: "2ef4969c2f9db610584640ca9c0f526f"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.155362 17397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:31:25.159368 17412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.161516 17397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:25.161819 17397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "8650ef1ba2d24234981a633a633b260a"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "13a7bc08480852a6fef9e6dbec8ac468"
server_key_iv: "2ef4969c2f9db610584640ca9c0f526f"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.162026 17397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:25.178313 17397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:25.182924 17397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:25.183182 17397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:25.192574 17397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37635
I20260925 16:31:25.192611 17464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37635 every 8 connection(s)
I20260925 16:31:25.193980 17397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:25.198181 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:25.199600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17397
I20260925 16:31:25.199755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:25.200093 392 external_mini_cluster.cc:1428] Setting key 398d96226222788cd4d3ccf1c6a0ee42
I20260925 16:31:25.206156 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: Bootstrap starting.
I20260925 16:31:25.208741 17465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:25.209908 17465 log.cc:824] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:31:25 dist-test-slave-31kt krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:25.212549 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: No bootstrap required, opened a new log
I20260925 16:31:25.217360 17465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8650ef1ba2d24234981a633a633b260a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37635 } }
I20260925 16:31:25.217815 17465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:25.217947 17465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8650ef1ba2d24234981a633a633b260a, State: Initialized, Role: FOLLOWER
I20260925 16:31:25.218484 17465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8650ef1ba2d24234981a633a633b260a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37635 } }
I20260925 16:31:25.218632 17465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:25.218700 17465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:25.218856 17465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:25.219164 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:25.202026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.98.62:37635)
0925 16:31:25.202703 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:25.202732 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:25.202788 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:31:25.203787 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:25.203814 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:25.204326 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:25.204794 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:25.205876 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.207089 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.208120 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.208751 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.208964 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:25.212662 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:25.212674 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:25.212703 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:25.212752 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:25.215122 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:25.215748 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:25.215755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:25.215762 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:25.215922 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:25.216241 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:25.216244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:25.216245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:25.216844 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:25.217101 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:25.217418 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 16:31:25.217894 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":197,"threads_started":1}
I20260925 16:31:25.220023 17465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8650ef1ba2d24234981a633a633b260a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37635 } }
I20260925 16:31:25.220515 17465 leader_election.cc:302] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8650ef1ba2d24234981a633a633b260a; no voters:
I20260925 16:31:25.220916 17465 leader_election.cc:288] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:25.221096 17470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:25.221407 17470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [term 1 LEADER]: Becoming Leader. State: Replica: 8650ef1ba2d24234981a633a633b260a, State: Running, Role: LEADER
I20260925 16:31:25.221935 17470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8650ef1ba2d24234981a633a633b260a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37635 } }
I20260925 16:31:25.222551 17465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:25.224063 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8650ef1ba2d24234981a633a633b260a. Latest consensus state: current_term: 1 leader_uuid: "8650ef1ba2d24234981a633a633b260a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8650ef1ba2d24234981a633a633b260a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37635 } } }
I20260925 16:31:25.224233 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:25.224644 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8650ef1ba2d24234981a633a633b260a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8650ef1ba2d24234981a633a633b260a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37635 } } }
I20260925 16:31:25.224745 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:25.228137 17485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:25.228246 17485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:25.228632 17482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:25.229422 17482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:25.237589 17482 catalog_manager.cc:1347] Generated new cluster ID: a3479ceca99b4d30a67f05a1153b8c7d
I20260925 16:31:25.237695 17482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:25.259380 17482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:25.261183 17482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:25.270430 17482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: Generated new TSK 0
I20260925 16:31:25.271646 17482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:31:25.310355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37635
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:25.439922 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:25.440363 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:25.440418 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:25.446682 17491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:25.446822 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:25.446905 17491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:25.454439 17491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:25.455997 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:25.457302 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:25.469998 17499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:25.470033 17496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:25.470021 17497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:25.470281 17491 server_base.cc:1034] running on GCE node
I20260925 16:31:25.470861 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:25.471832 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:25.473064 17491 hybrid_clock.cc:648] HybridClock initialized: now 1790353885473039 us; error 42 us; skew 500 ppm
Sep 25 16:31:25 dist-test-slave-31kt krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:25.476485 17491 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:31:25.477595 17491 webserver.cc:460] Webserver started at http://127.0.98.1:42585/ using document root <none> and password file <none>
I20260925 16:31:25.478425 17491 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:25.478557 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:25.478842 17491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:25.481242 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec5d6772be764e14b2f9bfc592163b8c"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "d4440183db01020d4d4ffe8491a17713"
server_key_iv: "27fc75f4ce1660664097e339db9ae237"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.481882 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec5d6772be764e14b2f9bfc592163b8c"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "d4440183db01020d4d4ffe8491a17713"
server_key_iv: "27fc75f4ce1660664097e339db9ae237"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.486780 17491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 16:31:25.490305 17506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.492251 17491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:31:25.492463 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "ec5d6772be764e14b2f9bfc592163b8c"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "d4440183db01020d4d4ffe8491a17713"
server_key_iv: "27fc75f4ce1660664097e339db9ae237"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.492630 17491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:25.516184 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:25.519588 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:25.519778 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:25.520392 17491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:25.521366 17491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:25.521420 17491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.521463 17491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:25.521497 17491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.531591 17491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44395
I20260925 16:31:25.531693 17619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44395 every 8 connection(s)
I20260925 16:31:25.532637 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:25.539973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17491
I20260925 16:31:25.540124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:31:25.540438 392 external_mini_cluster.cc:1428] Setting key fe6e2ba9f12b28276765d4aebb8b5d39
Sep 25 16:31:25 dist-test-slave-31kt krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:25.550418 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:25.535205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45045 (local address 127.0.98.62:37635)
0925 16:31:25.535419 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:25.535425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:25.536656 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 16:31:25.538149 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:25.538153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:25.538221 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:25.538301 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:25.539675 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.540292 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.541621 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.541934 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.542056 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:25.545416 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:25.545422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:25.545425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:25.545469 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:25.547514 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:25.548356 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:25.548361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:25.548363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:25.548487 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:25.548894 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:25.548897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:25.548899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:25.549177 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:25.549295 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:25.550059 (+ 764us) server_negotiation.cc:299] Negotiation successful
0925 16:31:25.550228 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:31:25.551605 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:25.535691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37635 (local address 127.0.98.1:45045)
0925 16:31:25.536407 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:25.536474 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:25.537873 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:25.538522 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:25.538530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:25.538956 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:25.539486 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:25.539503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.540423 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.540426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:25.541493 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:25.541504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.542201 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.542208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:25.542471 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:25.543250 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:25.543270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:25.545118 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:25.547766 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:25.547774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:25.547800 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:25.548213 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:25.548596 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:25.548598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:25.548599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:25.548785 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:25.549327 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:25.549334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:25.549707 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 16:31:25.549982 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 16:31:25.550375 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:31:25.553397 17620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37635
I20260925 16:31:25.553711 17620 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:25.554374 17620 heartbeater.cc:511] Master 127.0.98.62:37635 requested a full tablet report, sending...
I20260925 16:31:25.556362 17429 ts_manager.cc:194] Registered new tserver with Master: ec5d6772be764e14b2f9bfc592163b8c (127.0.98.1:44395)
I20260925 16:31:25.557708 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:31:25.569495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37635
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:25.694460 17625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:25.694785 17625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:25.694834 17625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:25.699335 17625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:25.699424 17625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:25.699494 17625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:25.705338 17625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:25.706784 17625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:25.708006 17625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:25.721006 17630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:25.720966 17633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:25.720969 17631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:25.721467 17625 server_base.cc:1034] running on GCE node
I20260925 16:31:25.722201 17625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:25.723181 17625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:25.726547 17625 hybrid_clock.cc:648] HybridClock initialized: now 1790353885726450 us; error 77 us; skew 500 ppm
Sep 25 16:31:25 dist-test-slave-31kt krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:25.737039 17625 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:31:25.738510 17625 webserver.cc:460] Webserver started at http://127.0.98.2:46119/ using document root <none> and password file <none>
I20260925 16:31:25.739267 17625 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:25.739346 17625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:25.739602 17625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:25.742241 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "be1ee5084f924a5b9ba96d1c106613e4"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "8f0623413c296fbda346b36d69f05559"
server_key_iv: "75cc2d66af1a94babcfc764cb2b00044"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.742931 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be1ee5084f924a5b9ba96d1c106613e4"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "8f0623413c296fbda346b36d69f05559"
server_key_iv: "75cc2d66af1a94babcfc764cb2b00044"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.748353 17625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 16:31:25.752032 17640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.753974 17625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:25.754160 17625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "be1ee5084f924a5b9ba96d1c106613e4"
format_stamp: "Formatted at 2026-09-25 16:31:25 on dist-test-slave-31kt"
server_key: "8f0623413c296fbda346b36d69f05559"
server_key_iv: "75cc2d66af1a94babcfc764cb2b00044"
server_key_version: "encryptionkey@0"
I20260925 16:31:25.754345 17625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:25.773048 17625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:25.777429 17625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:25.777661 17625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:25.778520 17625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:25.779654 17625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:25.779711 17625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.779759 17625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:25.779774 17625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:25.791867 17625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40451
I20260925 16:31:25.791882 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40451 every 8 connection(s)
I20260925 16:31:25.793147 17625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:25.798472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17625
I20260925 16:31:25.798606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:31:25.798910 392 external_mini_cluster.cc:1428] Setting key a52c096b16034597896c994743da7f73
Sep 25 16:31:25 dist-test-slave-31kt krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:25.809111 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:25.795650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40723 (local address 127.0.98.62:37635)
0925 16:31:25.795789 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:25.795792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:25.796965 (+ 1173us) server_negotiation.cc:407] Connection header received
0925 16:31:25.798361 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:25.798365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:25.798437 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:25.798526 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:25.800141 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.800770 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.802050 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.802274 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.802365 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:25.805230 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:25.805234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:25.805236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:25.805264 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:25.806857 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:25.807457 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:25.807460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:25.807461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:25.807542 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:25.807895 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:25.807898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:25.807900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:25.808116 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:25.808212 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:25.808788 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 16:31:25.808934 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:31:25.810117 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:25.796077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37635 (local address 127.0.98.2:40723)
0925 16:31:25.796733 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:25.796790 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:25.798046 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:25.798770 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:25.798779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:25.799305 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:25.799972 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:25.799992 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.800901 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.800905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:25.801914 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:25.801925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:25.802437 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:25.802441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:25.802628 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:25.803250 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:25.803267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:25.804962 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:25.806997 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:25.807004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:25.807024 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:25.807349 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:25.807644 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:25.807648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:25.807650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:25.807800 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:25.808223 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:25.808232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:25.808529 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 16:31:25.808723 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 16:31:25.809120 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":139,"threads_started":1}
I20260925 16:31:25.811924 17754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37635
I20260925 16:31:25.812250 17754 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:31:25.812850 17754 heartbeater.cc:511] Master 127.0.98.62:37635 requested a full tablet report, sending...
I20260925 16:31:25.814421 17429 ts_manager.cc:194] Registered new tserver with Master: be1ee5084f924a5b9ba96d1c106613e4 (127.0.98.2:40451)
I20260925 16:31:25.815250 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:31:25.830092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37635
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:25.976970 17759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:25.977406 17759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:25.977466 17759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:25.982210 17759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:25.982331 17759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:25.982402 17759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:25.988970 17759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:25.990758 17759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:25.992309 17759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:26.006343 17767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:26.006307 17764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:26.006317 17765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:26.006705 17759 server_base.cc:1034] running on GCE node
I20260925 16:31:26.007433 17759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:26.008467 17759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:26.009672 17759 hybrid_clock.cc:648] HybridClock initialized: now 1790353886009656 us; error 49 us; skew 500 ppm
Sep 25 16:31:26 dist-test-slave-31kt krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:26.013309 17759 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:31:26.014706 17759 webserver.cc:460] Webserver started at http://127.0.98.3:43047/ using document root <none> and password file <none>
I20260925 16:31:26.015488 17759 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:26.015569 17759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:26.015789 17759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:26.018674 17759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e0d25ed9851424f82e2800e29d92c76"
format_stamp: "Formatted at 2026-09-25 16:31:26 on dist-test-slave-31kt"
server_key: "bd02fa1ba84ab8f96d9b6f1a287ccd26"
server_key_iv: "c37c5da563e173006e5a8272ad47c542"
server_key_version: "encryptionkey@0"
I20260925 16:31:26.019412 17759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e0d25ed9851424f82e2800e29d92c76"
format_stamp: "Formatted at 2026-09-25 16:31:26 on dist-test-slave-31kt"
server_key: "bd02fa1ba84ab8f96d9b6f1a287ccd26"
server_key_iv: "c37c5da563e173006e5a8272ad47c542"
server_key_version: "encryptionkey@0"
I20260925 16:31:26.024993 17759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:26.029040 17774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:26.031112 17759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:31:26.031343 17759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "7e0d25ed9851424f82e2800e29d92c76"
format_stamp: "Formatted at 2026-09-25 16:31:26 on dist-test-slave-31kt"
server_key: "bd02fa1ba84ab8f96d9b6f1a287ccd26"
server_key_iv: "c37c5da563e173006e5a8272ad47c542"
server_key_version: "encryptionkey@0"
I20260925 16:31:26.031530 17759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:26.060231 17759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:26.065011 17759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:26.065265 17759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:26.066118 17759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:26.067281 17759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:26.067349 17759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:26.067401 17759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:26.067435 17759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:26.079221 17759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43095
I20260925 16:31:26.079263 17887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43095 every 8 connection(s)
I20260925 16:31:26.080442 17759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:26.090194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17759
I20260925 16:31:26.090375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:31:26.090744 392 external_mini_cluster.cc:1428] Setting key 9728d031826092d347b145300256e70c
Sep 25 16:31:26 dist-test-slave-31kt krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:26.099825 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.083237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56539 (local address 127.0.98.62:37635)
0925 16:31:26.083473 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.083477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.084927 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 16:31:26.086697 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.086701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.086777 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.086910 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:26.088702 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.089284 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.091416 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.091732 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.091856 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.095035 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:26.095039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:26.095041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:26.095072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:26.096803 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:26.097608 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:26.097612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:26.097613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:26.097706 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:26.098436 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:26.098440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:26.098441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:26.098702 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:26.098810 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.099448 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.099644 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 16:31:26.100791 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.083769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37635 (local address 127.0.98.3:56539)
0925 16:31:26.084667 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.084730 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:26.086310 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:26.087150 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:26.087161 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:26.087733 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:26.088450 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.088470 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.089506 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.089510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.091183 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.091203 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.091985 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.091992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.092261 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.092936 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:26.092955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:26.094783 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:26.096983 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:26.096993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:26.097019 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:26.097490 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:26.097827 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:26.097830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:26.097833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:26.098215 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:26.098833 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:26.098843 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:26.099174 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 16:31:26.099361 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 16:31:26.099773 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":237,"threads_started":1}
I20260925 16:31:26.102312 17888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37635
I20260925 16:31:26.102581 17888 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:26.103209 17888 heartbeater.cc:511] Master 127.0.98.62:37635 requested a full tablet report, sending...
I20260925 16:31:26.104496 17429 ts_manager.cc:194] Registered new tserver with Master: 7e0d25ed9851424f82e2800e29d92c76 (127.0.98.3:43095)
I20260925 16:31:26.105137 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56539
I20260925 16:31:26.105857 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:31:26.117573 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.108043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.0.98.62:37635)
0925 16:31:26.108189 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.108192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.108294 (+ 102us) server_negotiation.cc:407] Connection header received
0925 16:31:26.108433 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.108436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.108487 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.108556 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:26.109293 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.109980 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.111019 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.111299 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.111417 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.113349 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:26.113353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:26.113355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:26.113392 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:26.115139 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:26.115834 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:26.115837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:26.115839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:26.115960 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:26.116389 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:26.116392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:26.116394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:26.116716 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:26.116899 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.117222 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.117377 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 16:31:26.121479 17429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:31:26.125119 17429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:31:26.144573 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.137215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.98.2:40451)
0925 16:31:26.138071 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.138089 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.138122 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:31:26.138234 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.138240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.138520 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.139221 (+ 701us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:26.139422 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.140624 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.142760 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.143760 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.143890 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.144090 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.144275 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.144389 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":180,"threads_started":1}
I20260925 16:31:26.144570 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.136988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40451 (local address 127.0.0.1:51176)
0925 16:31:26.137797 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.137858 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:26.138003 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:26.138757 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:26.138765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:26.138808 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:26.139040 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.139050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.140770 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.140773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.142596 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.142609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.143914 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.143918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.144056 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.144100 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 16:31:26.144219 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 16:31:26.144388 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":46,"thread_start_us":130,"threads_started":1}
I20260925 16:31:26.145174 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.137980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43095 (local address 127.0.0.1:48052)
0925 16:31:26.138670 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.138681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:26.138760 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:26.140270 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:26.140273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:26.140293 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:26.140394 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.140400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.141727 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.141730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.143084 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.143101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.144577 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.144582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.144789 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.144817 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 16:31:26.144889 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 16:31:26.144991 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":101,"thread_start_us":138,"threads_started":1}
I20260925 16:31:26.145747 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.138500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.0.98.3:43095)
0925 16:31:26.139818 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.139832 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.139854 (+ 22us) server_negotiation.cc:407] Connection header received
0925 16:31:26.139932 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.139936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.140106 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.140261 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:26.140502 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.141566 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.143252 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.144180 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.144317 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.144495 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.145418 (+ 923us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.145566 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":124,"threads_started":1}
I20260925 16:31:26.146021 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.137432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.0.1:45712)
0925 16:31:26.138380 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.138394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:26.138479 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:26.139529 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:26.139532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:26.139555 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:26.139689 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.139695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.140980 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.140985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.143019 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.143028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.144577 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.144582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.145654 (+ 1072us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.145680 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:31:26.145749 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 16:31:26.145892 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":907,"thread_start_us":110,"threads_started":1}
I20260925 16:31:26.146612 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.138131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.0.98.1:44395)
0925 16:31:26.138992 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.138999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.139040 (+ 41us) server_negotiation.cc:407] Connection header received
0925 16:31:26.139137 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.139142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.139344 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.139506 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:26.139873 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.140828 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.143185 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.144183 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.144311 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.144495 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.146288 (+ 1793us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.146448 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":91,"threads_started":1}
I20260925 16:31:26.150110 17822 tablet_service.cc:1467] Processing CreateTablet for tablet 474752fbb7254600b8d861db79975704 (DEFAULT_TABLE table=test-table [id=06e654287a3e48d8ab1156933b7662b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:26.150112 17688 tablet_service.cc:1467] Processing CreateTablet for tablet 474752fbb7254600b8d861db79975704 (DEFAULT_TABLE table=test-table [id=06e654287a3e48d8ab1156933b7662b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:26.150249 17554 tablet_service.cc:1467] Processing CreateTablet for tablet 474752fbb7254600b8d861db79975704 (DEFAULT_TABLE table=test-table [id=06e654287a3e48d8ab1156933b7662b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:26.151835 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474752fbb7254600b8d861db79975704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:26.151839 17688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474752fbb7254600b8d861db79975704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:26.151865 17554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474752fbb7254600b8d861db79975704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:26.160523 17905 tablet_bootstrap.cc:492] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: Bootstrap starting.
I20260925 16:31:26.162420 17907 tablet_bootstrap.cc:492] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Bootstrap starting.
I20260925 16:31:26.162906 17906 tablet_bootstrap.cc:492] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Bootstrap starting.
I20260925 16:31:26.163828 17905 tablet_bootstrap.cc:654] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:26.164701 17905 log.cc:824] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:26.165544 17907 tablet_bootstrap.cc:654] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:26.165544 17906 tablet_bootstrap.cc:654] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:26.167086 17905 tablet_bootstrap.cc:492] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: No bootstrap required, opened a new log
I20260925 16:31:26.167217 17907 log.cc:824] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:26.167217 17906 log.cc:824] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:26.167315 17905 ts_tablet_manager.cc:1397] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:31:26.169971 17906 tablet_bootstrap.cc:492] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: No bootstrap required, opened a new log
I20260925 16:31:26.169970 17907 tablet_bootstrap.cc:492] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: No bootstrap required, opened a new log
I20260925 16:31:26.170224 17906 ts_tablet_manager.cc:1397] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:31:26.170224 17907 ts_tablet_manager.cc:1397] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 16:31:26.171679 17905 raft_consensus.cc:348] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.171963 17905 raft_consensus.cc:374] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:26.172019 17905 raft_consensus.cc:729] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5d6772be764e14b2f9bfc592163b8c, State: Initialized, Role: FOLLOWER
I20260925 16:31:26.172492 17905 consensus_queue.cc:260] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.173404 17620 heartbeater.cc:503] Master 127.0.98.62:37635 was elected leader, sending a full tablet report...
I20260925 16:31:26.173641 17905 ts_tablet_manager.cc:1428] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 16:31:26.174772 17907 raft_consensus.cc:348] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.175176 17907 raft_consensus.cc:374] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:26.175242 17907 raft_consensus.cc:729] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0d25ed9851424f82e2800e29d92c76, State: Initialized, Role: FOLLOWER
I20260925 16:31:26.175503 17906 raft_consensus.cc:348] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.175843 17906 raft_consensus.cc:374] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:26.175906 17906 raft_consensus.cc:729] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1ee5084f924a5b9ba96d1c106613e4, State: Initialized, Role: FOLLOWER
I20260925 16:31:26.175879 17907 consensus_queue.cc:260] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.176456 17906 consensus_queue.cc:260] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.176944 17888 heartbeater.cc:503] Master 127.0.98.62:37635 was elected leader, sending a full tablet report...
I20260925 16:31:26.177544 17907 ts_tablet_manager.cc:1428] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:31:26.177695 17906 ts_tablet_manager.cc:1428] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:31:26.178805 17754 heartbeater.cc:503] Master 127.0.98.62:37635 was elected leader, sending a full tablet report...
W20260925 16:31:26.284456 17621 tablet.cc:2369] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:31:26.294849 17755 tablet.cc:2369] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:31:26.331815 17889 tablet.cc:2369] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:31:26.433656 17913 raft_consensus.cc:482] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:26.434077 17913 raft_consensus.cc:504] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.435542 17913 leader_election.cc:288] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e0d25ed9851424f82e2800e29d92c76 (127.0.98.3:43095), ec5d6772be764e14b2f9bfc592163b8c (127.0.98.1:44395)
I20260925 16:31:26.441311 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.435844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43095 (local address 127.0.98.2:36267)
0925 16:31:26.436295 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.436315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:26.436531 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:26.437051 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:26.437054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:26.437075 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:26.437252 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.437258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.438561 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.438564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.439758 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.439778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.440669 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.440674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.440849 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.440883 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 16:31:26.440957 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 16:31:26.441097 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":102,"thread_start_us":121,"threads_started":1}
I20260925 16:31:26.441285 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.435982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36267 (local address 127.0.98.3:43095)
0925 16:31:26.436139 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.436144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.436411 (+ 267us) server_negotiation.cc:407] Connection header received
0925 16:31:26.436687 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.436692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.436775 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.436828 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:26.437442 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.438358 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.439946 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.440470 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.440630 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.440723 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.441007 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.441103 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:31:26.441285 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.436539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40913 (local address 127.0.98.1:44395)
0925 16:31:26.436701 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.436705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.436723 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:31:26.436789 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.436792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.436842 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.436916 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:26.437540 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.438358 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.439984 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.440511 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.440642 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.440720 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.441004 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.441108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:31:26.441318 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.435902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.98.2:40913)
0925 16:31:26.436488 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.436508 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:26.436693 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:26.437033 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:26.437036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:26.437058 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:26.437306 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.437317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.438571 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.438575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.439759 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:26.439778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.440671 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.440675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:26.440844 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.440879 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 16:31:26.440970 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 16:31:26.441100 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":132,"thread_start_us":133,"threads_started":1}
I20260925 16:31:26.442471 17574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474752fbb7254600b8d861db79975704" candidate_uuid: "be1ee5084f924a5b9ba96d1c106613e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5d6772be764e14b2f9bfc592163b8c" is_pre_election: true
I20260925 16:31:26.442474 17842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474752fbb7254600b8d861db79975704" candidate_uuid: "be1ee5084f924a5b9ba96d1c106613e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0d25ed9851424f82e2800e29d92c76" is_pre_election: true
I20260925 16:31:26.442886 17574 raft_consensus.cc:2454] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1ee5084f924a5b9ba96d1c106613e4 in term 0.
I20260925 16:31:26.442888 17842 raft_consensus.cc:2454] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1ee5084f924a5b9ba96d1c106613e4 in term 0.
I20260925 16:31:26.443552 17644 leader_election.cc:302] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1ee5084f924a5b9ba96d1c106613e4, ec5d6772be764e14b2f9bfc592163b8c; no voters:
I20260925 16:31:26.443989 17913 raft_consensus.cc:2781] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:31:26.444151 17913 raft_consensus.cc:482] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:31:26.444188 17913 raft_consensus.cc:3037] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:26.445492 17913 raft_consensus.cc:504] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.446035 17913 leader_election.cc:288] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7e0d25ed9851424f82e2800e29d92c76 (127.0.98.3:43095), ec5d6772be764e14b2f9bfc592163b8c (127.0.98.1:44395)
I20260925 16:31:26.446585 17842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474752fbb7254600b8d861db79975704" candidate_uuid: "be1ee5084f924a5b9ba96d1c106613e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e0d25ed9851424f82e2800e29d92c76"
I20260925 16:31:26.446761 17842 raft_consensus.cc:3037] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:26.446955 17574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474752fbb7254600b8d861db79975704" candidate_uuid: "be1ee5084f924a5b9ba96d1c106613e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5d6772be764e14b2f9bfc592163b8c"
I20260925 16:31:26.447180 17574 raft_consensus.cc:3037] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:26.448098 17842 raft_consensus.cc:2454] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1ee5084f924a5b9ba96d1c106613e4 in term 1.
I20260925 16:31:26.448567 17642 leader_election.cc:302] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0d25ed9851424f82e2800e29d92c76, be1ee5084f924a5b9ba96d1c106613e4; no voters:
I20260925 16:31:26.448669 17574 raft_consensus.cc:2454] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1ee5084f924a5b9ba96d1c106613e4 in term 1.
I20260925 16:31:26.448799 17913 raft_consensus.cc:2781] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:26.449055 17913 raft_consensus.cc:686] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 1 LEADER]: Becoming Leader. State: Replica: be1ee5084f924a5b9ba96d1c106613e4, State: Running, Role: LEADER
I20260925 16:31:26.449515 17913 consensus_queue.cc:237] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260925 16:31:26.453634 17429 catalog_manager.cc:5495] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to be1ee5084f924a5b9ba96d1c106613e4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "be1ee5084f924a5b9ba96d1c106613e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ee5084f924a5b9ba96d1c106613e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } health_report { overall_health: UNKNOWN } } }
I20260925 16:31:26.519997 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.515532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.0.98.2:40451)
0925 16:31:26.515683 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.515688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.515707 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:31:26.515914 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.515917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.515989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.516103 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:31:26.516371 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.516935 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.518108 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.518398 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.518508 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.518868 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:31:26.519517 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:31:26.519646 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.519722 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.519806 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:31:26.529340 17574 raft_consensus.cc:1261] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 1 FOLLOWER]: Refusing update from remote peer be1ee5084f924a5b9ba96d1c106613e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:26.529345 17842 raft_consensus.cc:1261] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 1 FOLLOWER]: Refusing update from remote peer be1ee5084f924a5b9ba96d1c106613e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:26.530282 17913 consensus_queue.cc:1035] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e0d25ed9851424f82e2800e29d92c76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:26.530642 17916 consensus_queue.cc:1035] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5d6772be764e14b2f9bfc592163b8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:26.545178 17920 mvcc.cc:205] Tried to move back new op lower bound from 7333289519212318720 to 7333289518900289536. Current Snapshot: MvccSnapshot[applied={T|T < 7333289519212318720}]
I20260925 16:31:26.545434 17921 mvcc.cc:205] Tried to move back new op lower bound from 7333289519212318720 to 7333289518900289536. Current Snapshot: MvccSnapshot[applied={T|T < 7333289519212318720}]
I20260925 16:31:26.555223 17428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:26.555476 17428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:26.559347 17428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8650ef1ba2d24234981a633a633b260a: Sending DeleteTablet for 3 replicas of tablet 474752fbb7254600b8d861db79975704
I20260925 16:31:26.560738 17554 tablet_service.cc:1514] Processing DeleteTablet for tablet 474752fbb7254600b8d861db79975704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45712
I20260925 16:31:26.560910 17688 tablet_service.cc:1514] Processing DeleteTablet for tablet 474752fbb7254600b8d861db79975704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51176
I20260925 16:31:26.561455 17929 tablet_replica.cc:330] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c: stopping tablet replica
I20260925 16:31:26.561465 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet 474752fbb7254600b8d861db79975704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48052
I20260925 16:31:26.561988 17929 raft_consensus.cc:2229] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:26.562211 17930 tablet_replica.cc:330] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: stopping tablet replica
I20260925 16:31:26.562546 17929 raft_consensus.cc:2258] T 474752fbb7254600b8d861db79975704 P ec5d6772be764e14b2f9bfc592163b8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:26.562599 17930 raft_consensus.cc:2229] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:31:26.562776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17491
I20260925 16:31:26.564124 17930 raft_consensus.cc:2258] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:26.565548 17931 tablet_replica.cc:330] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: stopping tablet replica
I20260925 16:31:26.565995 17931 raft_consensus.cc:2229] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:26.566422 17931 raft_consensus.cc:2258] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:26.567629 17930 ts_tablet_manager.cc:1905] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:26.568234 17931 ts_tablet_manager.cc:1905] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:26.571290 17930 ts_tablet_manager.cc:1918] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:26.571290 17931 ts_tablet_manager.cc:1918] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:26.571421 17931 log.cc:1196] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/474752fbb7254600b8d861db79975704
I20260925 16:31:26.571606 17930 log.cc:1196] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/474752fbb7254600b8d861db79975704
I20260925 16:31:26.571767 17931 ts_tablet_manager.cc:1939] T 474752fbb7254600b8d861db79975704 P 7e0d25ed9851424f82e2800e29d92c76: Deleting consensus metadata
I20260925 16:31:26.572086 17930 ts_tablet_manager.cc:1939] T 474752fbb7254600b8d861db79975704 P be1ee5084f924a5b9ba96d1c106613e4: Deleting consensus metadata
W20260925 16:31:26.572561 17416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44395 (error 108)
I20260925 16:31:26.573522 17414 catalog_manager.cc:4841] TS 7e0d25ed9851424f82e2800e29d92c76 (127.0.98.3:43095): tablet 474752fbb7254600b8d861db79975704 (table test-table [id=06e654287a3e48d8ab1156933b7662b7]) successfully deleted
I20260925 16:31:26.573779 17414 catalog_manager.cc:4841] TS be1ee5084f924a5b9ba96d1c106613e4 (127.0.98.2:40451): tablet 474752fbb7254600b8d861db79975704 (table test-table [id=06e654287a3e48d8ab1156933b7662b7]) successfully deleted
I20260925 16:31:26.574616 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.574195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.0.1:55924)
0925 16:31:26.574359 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:26.574453 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 16:31:26.574599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17625
W20260925 16:31:26.575407 17416 catalog_manager.cc:4584] TS ec5d6772be764e14b2f9bfc592163b8c (127.0.98.1:44395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 474752fbb7254600b8d861db79975704: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44395: connect: Connection refused (error 111)
I20260925 16:31:26.582226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17759
I20260925 16:31:26.588964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17397
2026-09-25T16:31:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2751 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T16:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:26Z Disabled control of system clock
I20260925 16:31:26.624173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:31:26.766957 17936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:26.767413 17936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:26.767457 17936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:26.772156 17936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:26.772255 17936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:26.772270 17936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:26.772289 17936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:26.772303 17936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:26.778540 17936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:26.780432 17936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:26.782701 17936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:26.794764 17941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:26.794756 17942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:26.794752 17944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:26.795254 17936 server_base.cc:1034] running on GCE node
I20260925 16:31:26.796161 17936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:26.797636 17936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:26.798854 17936 hybrid_clock.cc:648] HybridClock initialized: now 1790353886798822 us; error 59 us; skew 500 ppm
I20260925 16:31:26.801960 17936 webserver.cc:460] Webserver started at http://127.0.98.62:33517/ using document root <none> and password file <none>
I20260925 16:31:26.802790 17936 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:26.802872 17936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:26.803112 17936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:26.806072 17936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "395343bd8d4d43e59119bf4f7fa6170a"
format_stamp: "Formatted at 2026-09-25 16:31:26 on dist-test-slave-31kt"
server_key: "8754cc60f775d5f0b393d109d6979b49"
server_key_iv: "589665330d375c0b2537d06565a2368c"
server_key_version: "encryptionkey@0"
I20260925 16:31:26.806784 17936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "395343bd8d4d43e59119bf4f7fa6170a"
format_stamp: "Formatted at 2026-09-25 16:31:26 on dist-test-slave-31kt"
server_key: "8754cc60f775d5f0b393d109d6979b49"
server_key_iv: "589665330d375c0b2537d06565a2368c"
server_key_version: "encryptionkey@0"
I20260925 16:31:26.812021 17936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:31:26.816262 17950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:26.818279 17936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:26.818495 17936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "395343bd8d4d43e59119bf4f7fa6170a"
format_stamp: "Formatted at 2026-09-25 16:31:26 on dist-test-slave-31kt"
server_key: "8754cc60f775d5f0b393d109d6979b49"
server_key_iv: "589665330d375c0b2537d06565a2368c"
server_key_version: "encryptionkey@0"
I20260925 16:31:26.818691 17936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:26.830987 17936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:26.831925 17936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:26.832135 17936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:26.841435 17936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35695
I20260925 16:31:26.841559 18002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35695 every 8 connection(s)
I20260925 16:31:26.842795 17936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:26.843107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 17936
I20260925 16:31:26.843251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:26.843623 392 external_mini_cluster.cc:1428] Setting key ad7ee64add5fffda99b9fb23fcbdb163
I20260925 16:31:26.847433 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:26.855773 18003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: Bootstrap starting.
I20260925 16:31:26.858009 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:26.845605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.0.98.62:35695)
0925 16:31:26.846622 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:26.846644 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:26.846699 (+ 55us) server_negotiation.cc:407] Connection header received
0925 16:31:26.847659 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:26.847685 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:26.848249 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:26.848678 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:26.849195 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.850763 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.852428 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:26.853591 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:26.853893 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:26.855169 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:26.855180 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:26.855208 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:26.855899 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:26.856024 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:26.856064 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:26.856195 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:26.856420 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 16:31:26.856851 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":196,"threads_started":1}
I20260925 16:31:26.858572 18003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:26.859468 18003 log.cc:824] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:26.862026 18003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: No bootstrap required, opened a new log
I20260925 16:31:26.865814 18003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395343bd8d4d43e59119bf4f7fa6170a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260925 16:31:26.866103 18003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:26.866139 18003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395343bd8d4d43e59119bf4f7fa6170a, State: Initialized, Role: FOLLOWER
I20260925 16:31:26.866577 18003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395343bd8d4d43e59119bf4f7fa6170a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260925 16:31:26.866714 18003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:26.866755 18003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:26.866842 18003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:26.867882 18003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395343bd8d4d43e59119bf4f7fa6170a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260925 16:31:26.868238 18003 leader_election.cc:302] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395343bd8d4d43e59119bf4f7fa6170a; no voters:
I20260925 16:31:26.868566 18003 leader_election.cc:288] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:26.868883 18008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:26.869221 18008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [term 1 LEADER]: Becoming Leader. State: Replica: 395343bd8d4d43e59119bf4f7fa6170a, State: Running, Role: LEADER
I20260925 16:31:26.869655 18008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395343bd8d4d43e59119bf4f7fa6170a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260925 16:31:26.870030 18003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:26.871920 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395343bd8d4d43e59119bf4f7fa6170a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395343bd8d4d43e59119bf4f7fa6170a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260925 16:31:26.872164 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:26.871942 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395343bd8d4d43e59119bf4f7fa6170a. Latest consensus state: current_term: 1 leader_uuid: "395343bd8d4d43e59119bf4f7fa6170a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395343bd8d4d43e59119bf4f7fa6170a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260925 16:31:26.872617 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:26.875793 18023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:26.875896 18023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:26.875967 18022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:26.876732 18022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:26.884517 18022 catalog_manager.cc:1347] Generated new cluster ID: 302d9285b31e4b75a21596eff0fc1a21
I20260925 16:31:26.884624 18022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:26.911586 18022 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:26.913656 18022 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:26.932257 18022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: Generated new TSK 0
I20260925 16:31:26.933386 18022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:26.951926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--builtin_ntp_servers=127.0.98.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:31:27.115983 18027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:27.116386 18027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:27.116430 18027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:27.121181 18027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:27.121284 18027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:27.121354 18027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:27.127409 18027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:27.129004 18027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:27.130920 18027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:27.143322 18032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:27.143322 18033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:27.143337 18035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:27.143783 18027 server_base.cc:1034] running on GCE node
I20260925 16:31:27.144335 18027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:27.145234 18027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:27.146417 18027 hybrid_clock.cc:648] HybridClock initialized: now 1790353887146378 us; error 62 us; skew 500 ppm
I20260925 16:31:27.148886 18027 webserver.cc:460] Webserver started at http://127.0.98.1:36639/ using document root <none> and password file <none>
I20260925 16:31:27.149623 18027 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:27.149699 18027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:27.149981 18027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:27.152732 18027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "31f738aa0c658ffc2fd1f4b10da09b5c"
server_key_iv: "9851f6567515e3cef91c6bfdeddd2b9a"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.153465 18027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "31f738aa0c658ffc2fd1f4b10da09b5c"
server_key_iv: "9851f6567515e3cef91c6bfdeddd2b9a"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.159041 18027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:31:27.163223 18041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.165321 18027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260925 16:31:27.165555 18027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "31f738aa0c658ffc2fd1f4b10da09b5c"
server_key_iv: "9851f6567515e3cef91c6bfdeddd2b9a"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.165743 18027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:27.187770 18027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:27.188763 18027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:27.188983 18027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:27.189780 18027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:27.191116 18027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:27.191175 18027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.191224 18027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:27.191238 18027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.203153 18027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36705
I20260925 16:31:27.203266 18154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36705 every 8 connection(s)
I20260925 16:31:27.204439 18027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:27.213115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18027
I20260925 16:31:27.213287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:31:27.213671 392 external_mini_cluster.cc:1428] Setting key 1bdd1280264fa5d605fbde9b278ab176
I20260925 16:31:27.216532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--builtin_ntp_servers=127.0.98.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:31:27.217962 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.207056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51759 (local address 127.0.98.62:35695)
0925 16:31:27.207234 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.207238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.208829 (+ 1591us) server_negotiation.cc:407] Connection header received
0925 16:31:27.210452 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.210456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.210535 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.210708 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:27.211880 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.212578 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.214772 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.215098 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.215230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.217228 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:27.217233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:27.217235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:27.217253 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:27.217282 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:27.217291 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:27.217364 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.217646 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.217799 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 16:31:27.219635 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.207587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:51759)
0925 16:31:27.208597 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.208664 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.210162 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.210979 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.210987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.211091 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:27.211673 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.211688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.212829 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.212835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.214580 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.214600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.215353 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.215359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.215709 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.216666 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:27.216669 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:27.216798 (+ 129us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:27.216963 (+ 165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:27.217387 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:27.217397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:27.217404 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.217578 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.218124 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":195,"threads_started":1}
I20260925 16:31:27.221406 18155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260925 16:31:27.221709 18155 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:27.222414 18155 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260925 16:31:27.224445 17967 ts_manager.cc:194] Registered new tserver with Master: 10a27b9a19b24d42b1cd9adac4cc2ca3 (127.0.98.1:36705)
I20260925 16:31:27.225832 17967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51759
W20260925 16:31:27.357211 18158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:27.357647 18158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:27.357713 18158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:27.363226 18158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:27.363343 18158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:27.363442 18158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:27.369495 18158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:27.371330 18158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:27.373454 18158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:27.386257 18163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:27.386240 18166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:27.386240 18164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:27.386351 18158 server_base.cc:1034] running on GCE node
I20260925 16:31:27.387143 18158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:27.387996 18158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:27.389389 18158 hybrid_clock.cc:648] HybridClock initialized: now 1790353887389333 us; error 67 us; skew 500 ppm
I20260925 16:31:27.392619 18158 webserver.cc:460] Webserver started at http://127.0.98.2:33079/ using document root <none> and password file <none>
I20260925 16:31:27.393368 18158 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:27.393462 18158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:27.393690 18158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:27.396256 18158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "68345ecf91de48cf81fa3afa6a08ed76"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "39819d7cd7f7e89f1a18c5b97a78a575"
server_key_iv: "48c5ce6bf87e3146a17ba0abd5c9d68d"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.396894 18158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "68345ecf91de48cf81fa3afa6a08ed76"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "39819d7cd7f7e89f1a18c5b97a78a575"
server_key_iv: "48c5ce6bf87e3146a17ba0abd5c9d68d"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.401855 18158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:31:27.405334 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.407337 18158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:27.407516 18158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "68345ecf91de48cf81fa3afa6a08ed76"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "39819d7cd7f7e89f1a18c5b97a78a575"
server_key_iv: "48c5ce6bf87e3146a17ba0abd5c9d68d"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.407711 18158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:27.421453 18158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:27.422438 18158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:27.422672 18158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:27.423470 18158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:27.424758 18158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:27.424811 18158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.424861 18158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:27.424875 18158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.435956 18158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35617
I20260925 16:31:27.436055 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35617 every 8 connection(s)
I20260925 16:31:27.437261 18158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:27.446892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18158
I20260925 16:31:27.447023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:31:27.447285 392 external_mini_cluster.cc:1428] Setting key 13abb756fdddc2b53032ef9350528f5f
I20260925 16:31:27.449307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--builtin_ntp_servers=127.0.98.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:31:27.450531 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.439911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50239 (local address 127.0.98.62:35695)
0925 16:31:27.440157 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.440162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.441637 (+ 1475us) server_negotiation.cc:407] Connection header received
0925 16:31:27.443723 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.443727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.443808 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.443925 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:27.445549 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.446191 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.447941 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.448200 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.448308 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.449883 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:27.449888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:27.449890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:27.449909 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:27.449936 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:27.449945 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:27.450016 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.450212 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.450321 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 16:31:27.451848 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.440312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:50239)
0925 16:31:27.441371 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.441439 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.443319 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.444255 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.444266 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.444403 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:27.445217 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.445237 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.446402 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.446406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.447745 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.447763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.448397 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.448402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.448634 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.449427 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:27.449429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:27.449512 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:27.449640 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:27.450027 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:27.450039 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:27.450047 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.450212 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.450680 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":187,"threads_started":1}
I20260925 16:31:27.453843 18286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260925 16:31:27.454260 18286 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:27.455178 18286 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260925 16:31:27.457116 17967 ts_manager.cc:194] Registered new tserver with Master: 68345ecf91de48cf81fa3afa6a08ed76 (127.0.98.2:35617)
I20260925 16:31:27.458045 17967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50239
W20260925 16:31:27.589746 18289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:27.590197 18289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:27.590246 18289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:27.595366 18289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:27.595479 18289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:27.595548 18289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:27.601408 18289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:27.603015 18289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:27.604853 18289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:27.617326 18294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:27.617442 18297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:27.617641 18295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:27.617997 18289 server_base.cc:1034] running on GCE node
I20260925 16:31:27.618551 18289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:27.619378 18289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:27.620618 18289 hybrid_clock.cc:648] HybridClock initialized: now 1790353887620595 us; error 47 us; skew 500 ppm
I20260925 16:31:27.623929 18289 webserver.cc:460] Webserver started at http://127.0.98.3:37089/ using document root <none> and password file <none>
I20260925 16:31:27.625169 18289 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:27.625332 18289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:27.625592 18289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:27.628193 18289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b06c6af87a24f37bc0cd2291639c106"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "83cb8d0d4d6adb4e07c9d2396e4f4325"
server_key_iv: "45418595f3507890c00ec68e294dcba1"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.628862 18289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b06c6af87a24f37bc0cd2291639c106"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "83cb8d0d4d6adb4e07c9d2396e4f4325"
server_key_iv: "45418595f3507890c00ec68e294dcba1"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.633590 18289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 16:31:27.637535 18303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.639789 18289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:27.640105 18289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "4b06c6af87a24f37bc0cd2291639c106"
format_stamp: "Formatted at 2026-09-25 16:31:27 on dist-test-slave-31kt"
server_key: "83cb8d0d4d6adb4e07c9d2396e4f4325"
server_key_iv: "45418595f3507890c00ec68e294dcba1"
server_key_version: "encryptionkey@0"
I20260925 16:31:27.640280 18289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:27.679484 18289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:27.680335 18289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:27.680554 18289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:27.681252 18289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:27.682519 18289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:27.682578 18289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.682636 18289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:27.682665 18289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:27.693907 18289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37239
I20260925 16:31:27.693980 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37239 every 8 connection(s)
I20260925 16:31:27.695767 18289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:27.699577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18289
I20260925 16:31:27.699707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:31:27.700038 392 external_mini_cluster.cc:1428] Setting key a9e1a7276740f1642de3f8134465690f
I20260925 16:31:27.708564 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.698904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40265 (local address 127.0.98.62:35695)
0925 16:31:27.699128 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.699133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.700606 (+ 1473us) server_negotiation.cc:407] Connection header received
0925 16:31:27.702411 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.702415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.702500 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.702576 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:27.703695 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.704235 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.706034 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.706327 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.706454 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.707979 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:27.707983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:27.707985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:27.708006 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:27.708038 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:27.708047 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:27.708126 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.708295 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.708388 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 16:31:27.709681 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.699398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:40265)
0925 16:31:27.700389 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.700450 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.702165 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.702798 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.702805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.702928 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:27.703523 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.703539 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.704365 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.704369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.705860 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.705876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.706573 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.706577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.706883 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.707613 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:27.707614 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:27.707665 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:27.707750 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:27.708154 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:27.708162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:27.708167 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.708286 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.708597 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":212,"threads_started":1}
I20260925 16:31:27.711741 18417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260925 16:31:27.712118 18417 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:27.712967 18417 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260925 16:31:27.714485 17967 ts_manager.cc:194] Registered new tserver with Master: 4b06c6af87a24f37bc0cd2291639c106 (127.0.98.3:37239)
I20260925 16:31:27.715325 17967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40265
I20260925 16:31:27.715612 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:31:27.723893 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.718788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.98.62:35695)
0925 16:31:27.719017 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.719022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.719245 (+ 223us) server_negotiation.cc:407] Connection header received
0925 16:31:27.719398 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.719401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.719465 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.719541 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:27.719890 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.720573 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.721698 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.722067 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.722203 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.723314 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:27.723318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:27.723321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:27.723339 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:27.723370 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:27.723389 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:27.723484 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.723644 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.723721 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 16:31:27.728138 17967 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:31:27.732218 17967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:31:27.751283 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.744221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36705 (local address 127.0.0.1:37836)
0925 16:31:27.745579 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.745656 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.745864 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.746660 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.746667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.746708 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:27.747000 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.747009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.748106 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.748110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.749395 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.749406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.750575 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.750577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.750681 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.750773 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.750896 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.751077 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":59,"spinlock_wait_cycles":7424,"thread_start_us":163,"threads_started":1}
I20260925 16:31:27.751330 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.744587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.0.98.1:36705)
0925 16:31:27.745207 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.745214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.746046 (+ 832us) server_negotiation.cc:407] Connection header received
0925 16:31:27.746121 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.746126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.746322 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.746473 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:27.747146 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.747971 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.749571 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.750426 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.750555 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.750809 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.750999 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.751173 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":97,"threads_started":1}
I20260925 16:31:27.752326 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.745038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.98.2:35617)
0925 16:31:27.745855 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.745868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.746110 (+ 242us) server_negotiation.cc:407] Connection header received
0925 16:31:27.746371 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.746377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.746644 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.746768 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:27.747205 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.748448 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.750281 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.751121 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.751240 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.751618 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.752001 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.752142 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":104,"threads_started":1}
I20260925 16:31:27.753105 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.744943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.0.98.3:37239)
0925 16:31:27.745402 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.745524 (+ 122us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.745837 (+ 313us) server_negotiation.cc:407] Connection header received
0925 16:31:27.746078 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.746083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.746322 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.746473 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:27.747130 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.748479 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.750927 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.752068 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.752176 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.752360 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.752820 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.752938 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":188,"threads_started":1}
I20260925 16:31:27.753111 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.744430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35617 (local address 127.0.0.1:45604)
0925 16:31:27.746005 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.746020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.746190 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.746945 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.746948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.746963 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:27.747085 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.747091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.748600 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.748603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.750098 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.750109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.751700 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.751704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.751812 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.751843 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.751934 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.752046 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":116,"thread_start_us":128,"threads_started":1}
I20260925 16:31:27.753394 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.744154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37239 (local address 127.0.0.1:50888)
0925 16:31:27.745573 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.745656 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.745897 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.746660 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.746667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.746709 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:27.746999 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.747008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.748600 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.748603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.750096 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.750109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.752430 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.752434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.752547 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.752570 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.752640 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.752735 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":202,"spinlock_wait_cycles":8448,"thread_start_us":129,"threads_started":1}
I20260925 16:31:27.755736 18089 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4d8bfee347418fbeda1fd59abd66d6 (DEFAULT_TABLE table=test-table [id=8ce2ab6ecaaf4b77b4a169a8f5ab9f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:27.756417 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4d8bfee347418fbeda1fd59abd66d6 (DEFAULT_TABLE table=test-table [id=8ce2ab6ecaaf4b77b4a169a8f5ab9f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:27.756398 18351 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4d8bfee347418fbeda1fd59abd66d6 (DEFAULT_TABLE table=test-table [id=8ce2ab6ecaaf4b77b4a169a8f5ab9f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:27.757283 18089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4d8bfee347418fbeda1fd59abd66d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:27.757347 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4d8bfee347418fbeda1fd59abd66d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:27.757689 18351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4d8bfee347418fbeda1fd59abd66d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:27.765264 18434 tablet_bootstrap.cc:492] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Bootstrap starting.
I20260925 16:31:27.767629 18435 tablet_bootstrap.cc:492] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3: Bootstrap starting.
I20260925 16:31:27.768610 18434 tablet_bootstrap.cc:654] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:27.769618 18434 log.cc:824] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:27.770367 18435 tablet_bootstrap.cc:654] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:27.770915 18436 tablet_bootstrap.cc:492] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Bootstrap starting.
I20260925 16:31:27.771725 18435 log.cc:824] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:27.774240 18436 tablet_bootstrap.cc:654] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:27.774384 18434 tablet_bootstrap.cc:492] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: No bootstrap required, opened a new log
I20260925 16:31:27.774395 18435 tablet_bootstrap.cc:492] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3: No bootstrap required, opened a new log
I20260925 16:31:27.774663 18435 ts_tablet_manager.cc:1397] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:31:27.774663 18434 ts_tablet_manager.cc:1397] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 16:31:27.775502 18436 log.cc:824] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:27.778905 18436 tablet_bootstrap.cc:492] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: No bootstrap required, opened a new log
I20260925 16:31:27.779229 18436 ts_tablet_manager.cc:1397] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20260925 16:31:27.779706 18435 raft_consensus.cc:348] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.779717 18434 raft_consensus.cc:348] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.780073 18435 raft_consensus.cc:374] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:27.780073 18434 raft_consensus.cc:374] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:27.780150 18434 raft_consensus.cc:729] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06c6af87a24f37bc0cd2291639c106, State: Initialized, Role: FOLLOWER
I20260925 16:31:27.780149 18435 raft_consensus.cc:729] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a27b9a19b24d42b1cd9adac4cc2ca3, State: Initialized, Role: FOLLOWER
I20260925 16:31:27.780815 18435 consensus_queue.cc:260] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.781124 18434 consensus_queue.cc:260] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.781900 18155 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260925 16:31:27.782684 18434 ts_tablet_manager.cc:1428] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260925 16:31:27.783224 18417 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260925 16:31:27.783948 18435 ts_tablet_manager.cc:1428] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260925 16:31:27.784673 18436 raft_consensus.cc:348] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.784999 18436 raft_consensus.cc:374] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:27.785076 18436 raft_consensus.cc:729] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68345ecf91de48cf81fa3afa6a08ed76, State: Initialized, Role: FOLLOWER
I20260925 16:31:27.785629 18436 consensus_queue.cc:260] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.786790 18436 ts_tablet_manager.cc:1428] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260925 16:31:27.787043 18286 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260925 16:31:27.830667 18442 raft_consensus.cc:482] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:27.830950 18442 raft_consensus.cc:504] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.832527 18442 leader_election.cc:288] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b06c6af87a24f37bc0cd2291639c106 (127.0.98.3:37239), 10a27b9a19b24d42b1cd9adac4cc2ca3 (127.0.98.1:36705)
I20260925 16:31:27.837715 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.832963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37501 (local address 127.0.98.3:37239)
0925 16:31:27.833151 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.833157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.833503 (+ 346us) server_negotiation.cc:407] Connection header received
0925 16:31:27.833581 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.833584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.833637 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.833706 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:27.834168 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.834796 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.836285 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.836733 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.836820 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.836893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.837421 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.837530 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:31:27.837812 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.833097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43907 (local address 127.0.98.1:36705)
0925 16:31:27.833300 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.833305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.833591 (+ 286us) server_negotiation.cc:407] Connection header received
0925 16:31:27.833822 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.833826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.833886 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.833995 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:27.834432 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.835097 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.836239 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.836991 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.837166 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.837328 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.837536 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.837625 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:31:27.837813 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.832832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37239 (local address 127.0.98.2:37501)
0925 16:31:27.833258 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.833275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.833399 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.833829 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.833832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.833855 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:27.834020 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.834030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.834962 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.834967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.836143 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.836156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.836944 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.836950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.837118 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.837155 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.837469 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.837617 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":33}
I20260925 16:31:27.837818 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.832992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36705 (local address 127.0.98.2:43907)
0925 16:31:27.833440 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.833453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:27.833569 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:27.834041 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:27.834043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:27.834062 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:27.834225 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.834232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.835232 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.835236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.836111 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:27.836122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.837212 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.837217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:27.837329 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.837354 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 16:31:27.837445 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 16:31:27.837529 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":28,"thread_start_us":157,"threads_started":1}
I20260925 16:31:27.838863 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4d8bfee347418fbeda1fd59abd66d6" candidate_uuid: "68345ecf91de48cf81fa3afa6a08ed76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" is_pre_election: true
I20260925 16:31:27.838863 18371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4d8bfee347418fbeda1fd59abd66d6" candidate_uuid: "68345ecf91de48cf81fa3afa6a08ed76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06c6af87a24f37bc0cd2291639c106" is_pre_election: true
I20260925 16:31:27.839250 18109 raft_consensus.cc:2454] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68345ecf91de48cf81fa3afa6a08ed76 in term 0.
I20260925 16:31:27.839250 18371 raft_consensus.cc:2454] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68345ecf91de48cf81fa3afa6a08ed76 in term 0.
I20260925 16:31:27.839972 18173 leader_election.cc:302] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b06c6af87a24f37bc0cd2291639c106, 68345ecf91de48cf81fa3afa6a08ed76; no voters:
I20260925 16:31:27.840333 18442 raft_consensus.cc:2781] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:31:27.840478 18442 raft_consensus.cc:482] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:31:27.840543 18442 raft_consensus.cc:3037] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:27.842110 18442 raft_consensus.cc:504] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.842674 18442 leader_election.cc:288] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [CANDIDATE]: Term 1 election: Requested vote from peers 4b06c6af87a24f37bc0cd2291639c106 (127.0.98.3:37239), 10a27b9a19b24d42b1cd9adac4cc2ca3 (127.0.98.1:36705)
I20260925 16:31:27.843321 18371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4d8bfee347418fbeda1fd59abd66d6" candidate_uuid: "68345ecf91de48cf81fa3afa6a08ed76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06c6af87a24f37bc0cd2291639c106"
I20260925 16:31:27.843386 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4d8bfee347418fbeda1fd59abd66d6" candidate_uuid: "68345ecf91de48cf81fa3afa6a08ed76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3"
I20260925 16:31:27.843526 18371 raft_consensus.cc:3037] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:27.843546 18109 raft_consensus.cc:3037] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:27.845408 18109 raft_consensus.cc:2454] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68345ecf91de48cf81fa3afa6a08ed76 in term 1.
I20260925 16:31:27.845408 18371 raft_consensus.cc:2454] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68345ecf91de48cf81fa3afa6a08ed76 in term 1.
I20260925 16:31:27.846127 18173 leader_election.cc:302] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10a27b9a19b24d42b1cd9adac4cc2ca3, 4b06c6af87a24f37bc0cd2291639c106, 68345ecf91de48cf81fa3afa6a08ed76; no voters:
I20260925 16:31:27.846457 18442 raft_consensus.cc:2781] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:27.846825 18442 raft_consensus.cc:686] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 1 LEADER]: Becoming Leader. State: Replica: 68345ecf91de48cf81fa3afa6a08ed76, State: Running, Role: LEADER
I20260925 16:31:27.847221 18442 consensus_queue.cc:237] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } }
I20260925 16:31:27.851709 17965 catalog_manager.cc:5495] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68345ecf91de48cf81fa3afa6a08ed76 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "68345ecf91de48cf81fa3afa6a08ed76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68345ecf91de48cf81fa3afa6a08ed76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } health_report { overall_health: HEALTHY } } }
I20260925 16:31:27.883579 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.878811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.98.2:35617)
0925 16:31:27.879013 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:27.879018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:27.879037 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:31:27.879152 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:27.879155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:27.879239 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:27.879367 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:31:27.879789 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.880379 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.881389 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:27.881627 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:27.881732 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:27.882189 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:31:27.882846 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:31:27.882970 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:27.883182 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 16:31:27.883286 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 16:31:27.891887 18371 raft_consensus.cc:1261] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 1 FOLLOWER]: Refusing update from remote peer 68345ecf91de48cf81fa3afa6a08ed76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:27.891958 18109 raft_consensus.cc:1261] T 2c4d8bfee347418fbeda1fd59abd66d6 P 10a27b9a19b24d42b1cd9adac4cc2ca3 [term 1 FOLLOWER]: Refusing update from remote peer 68345ecf91de48cf81fa3afa6a08ed76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:27.892855 18444 consensus_queue.cc:1035] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b06c6af87a24f37bc0cd2291639c106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:27.893112 18442 consensus_queue.cc:1035] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a27b9a19b24d42b1cd9adac4cc2ca3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:27.902287 18448 mvcc.cc:205] Tried to move back new op lower bound from 7333289524795002880 to 7333289524625096704. Current Snapshot: MvccSnapshot[applied={T|T < 7333289524795002880}]
I20260925 16:31:27.904081 18447 mvcc.cc:205] Tried to move back new op lower bound from 7333289524795002880 to 7333289524625096704. Current Snapshot: MvccSnapshot[applied={T|T < 7333289524795002880 or (T in {7333289524795002880})}]
I20260925 16:31:27.915148 17966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:27.915410 17966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:27.918845 17966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 395343bd8d4d43e59119bf4f7fa6170a: Sending DeleteTablet for 3 replicas of tablet 2c4d8bfee347418fbeda1fd59abd66d6
I20260925 16:31:27.919937 18351 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4d8bfee347418fbeda1fd59abd66d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:27 UTC) from {username='slave'} at 127.0.0.1:50888
I20260925 16:31:27.919962 18089 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4d8bfee347418fbeda1fd59abd66d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:27 UTC) from {username='slave'} at 127.0.0.1:37836
I20260925 16:31:27.919962 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4d8bfee347418fbeda1fd59abd66d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:27 UTC) from {username='slave'} at 127.0.0.1:45604
I20260925 16:31:27.920790 18459 tablet_replica.cc:330] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: stopping tablet replica
I20260925 16:31:27.921224 18459 raft_consensus.cc:2229] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:31:27.921794 18459 raft_consensus.cc:2258] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:27.921873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18027
I20260925 16:31:27.924311 18459 ts_tablet_manager.cc:1905] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:27.927491 18459 ts_tablet_manager.cc:1918] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:27.927634 18459 log.cc:1196] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/2c4d8bfee347418fbeda1fd59abd66d6
I20260925 16:31:27.927971 18459 ts_tablet_manager.cc:1939] T 2c4d8bfee347418fbeda1fd59abd66d6 P 68345ecf91de48cf81fa3afa6a08ed76: Deleting consensus metadata
I20260925 16:31:27.929566 17953 catalog_manager.cc:4841] TS 68345ecf91de48cf81fa3afa6a08ed76 (127.0.98.2:35617): tablet 2c4d8bfee347418fbeda1fd59abd66d6 (table test-table [id=8ce2ab6ecaaf4b77b4a169a8f5ab9f36]) successfully deleted
I20260925 16:31:27.929895 18457 tablet_replica.cc:330] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: stopping tablet replica
I20260925 16:31:27.930315 18457 raft_consensus.cc:2229] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:27.930706 18457 raft_consensus.cc:2258] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 16:31:27.932199 17952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36705 (error 108)
I20260925 16:31:27.932363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18158
I20260925 16:31:27.932638 18457 ts_tablet_manager.cc:1905] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:27.935034 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:27.934566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36705 (local address 127.0.0.1:37840)
0925 16:31:27.934798 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:27.934916 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 16:31:27.935323 17952 catalog_manager.cc:4584] TS 10a27b9a19b24d42b1cd9adac4cc2ca3 (127.0.98.1:36705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c4d8bfee347418fbeda1fd59abd66d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36705: connect: Connection refused (error 111)
I20260925 16:31:27.935877 18457 ts_tablet_manager.cc:1918] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:27.935985 18457 log.cc:1196] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/2c4d8bfee347418fbeda1fd59abd66d6
I20260925 16:31:27.936285 18457 ts_tablet_manager.cc:1939] T 2c4d8bfee347418fbeda1fd59abd66d6 P 4b06c6af87a24f37bc0cd2291639c106: Deleting consensus metadata
I20260925 16:31:27.937913 17951 catalog_manager.cc:4841] TS 4b06c6af87a24f37bc0cd2291639c106 (127.0.98.3:37239): tablet 2c4d8bfee347418fbeda1fd59abd66d6 (table test-table [id=8ce2ab6ecaaf4b77b4a169a8f5ab9f36]) successfully deleted
I20260925 16:31:27.939744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18289
I20260925 16:31:27.945956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 17936
2026-09-25T16:31:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1356 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:28 dist-test-slave-31kt krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:28 dist-test-slave-31kt krb5kdc[18463](info): set up 2 sockets
Sep 25 16:31:28 dist-test-slave-31kt krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260925 16:31:28.934211 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:29.079527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:29.216092 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:29.216428 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:29.216475 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:29.220902 18476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:29.220988 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:29.221007 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:29.221027 18476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:29.221050 18476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:29.228511 18476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:29.229808 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:29.231271 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:29.242553 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:29.242553 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:29.242553 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:29.242973 18476 server_base.cc:1034] running on GCE node
I20260925 16:31:29.243973 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:29.246104 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:29.247447 18476 hybrid_clock.cc:648] HybridClock initialized: now 1790353889247423 us; error 57 us; skew 500 ppm
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:29.251057 18476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:29.252281 18476 webserver.cc:460] Webserver started at http://127.0.98.62:44095/ using document root <none> and password file <none>
I20260925 16:31:29.252947 18476 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:29.253026 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:29.253234 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:29.255625 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "f0599b9083e55254a1d817757981323b"
server_key_iv: "186c095386b2c4a3f31340d57b889ff9"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.256238 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "f0599b9083e55254a1d817757981323b"
server_key_iv: "186c095386b2c4a3f31340d57b889ff9"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.261037 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:31:29.265290 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.267366 18476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 16:31:29.267578 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "f0599b9083e55254a1d817757981323b"
server_key_iv: "186c095386b2c4a3f31340d57b889ff9"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.267752 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:29.289060 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:29.293140 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:29.293354 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:29.302405 18476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46881
I20260925 16:31:29.302439 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46881 every 8 connection(s)
I20260925 16:31:29.303831 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:29.307405 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:29.309286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18476
I20260925 16:31:29.309422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:29.309789 392 external_mini_cluster.cc:1428] Setting key da73b1baa9cf787e8bf23d5f53ab1811
I20260925 16:31:29.314946 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1: Bootstrap starting.
I20260925 16:31:29.318287 18544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:29.319402 18544 log.cc:824] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:29.323109 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1: No bootstrap required, opened a new log
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:29.328487 18544 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260925 16:31:29.328898 18544 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:29.328948 18544 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0fb67e1b9a437293b9baf1d6f7d3a1, State: Initialized, Role: FOLLOWER
I20260925 16:31:29.329646 18544 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260925 16:31:29.329878 18544 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:29.329995 18544 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:29.330127 18544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:29.331696 18544 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260925 16:31:29.332185 18544 leader_election.cc:302] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0fb67e1b9a437293b9baf1d6f7d3a1; no voters:
I20260925 16:31:29.332633 18544 leader_election.cc:288] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:29.332886 18549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:29.332960 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:29.311986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.0.98.62:46881)
0925 16:31:29.312677 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:29.312700 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:29.312756 (+ 56us) server_negotiation.cc:407] Connection header received
0925 16:31:29.313623 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:29.313649 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:29.314202 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:29.314634 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:29.315972 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.317994 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.319560 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.320618 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.320856 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:29.325136 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:29.325145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:29.325172 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:29.325227 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:29.328515 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:29.329330 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:29.329338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:29.329345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:29.329519 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:29.329852 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:29.329855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:29.329857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:29.330556 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:29.330788 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:29.331251 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 16:31:29.331657 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":196,"threads_started":1}
I20260925 16:31:29.333633 18549 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [term 1 LEADER]: Becoming Leader. State: Replica: bd0fb67e1b9a437293b9baf1d6f7d3a1, State: Running, Role: LEADER
I20260925 16:31:29.334249 18544 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:29.334210 18549 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260925 16:31:29.336371 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0fb67e1b9a437293b9baf1d6f7d3a1. Latest consensus state: current_term: 1 leader_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } } }
I20260925 16:31:29.336611 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:29.337306 18564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:29.336434 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0fb67e1b9a437293b9baf1d6f7d3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } } }
I20260925 16:31:29.337399 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:29.341662 18564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:29.350081 18564 catalog_manager.cc:1347] Generated new cluster ID: 4d88f9346e8d4b09b2a1b818ea47ba00
I20260925 16:31:29.350230 18564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:29.357507 18564 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:29.358943 18564 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:29.368534 18564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1: Generated new TSK 0
I20260925 16:31:29.369732 18564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:31:29.418419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46881
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:29.566890 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:29.567498 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:29.567564 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:29.572791 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:29.572957 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:29.573046 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:29.580200 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:29.581964 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:29.583484 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:29.597326 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:29.597326 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:29.597326 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:29.597718 18570 server_base.cc:1034] running on GCE node
I20260925 16:31:29.598512 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:29.600005 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:29.601351 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790353889601274 us; error 73 us; skew 500 ppm
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:29.605599 18570 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:31:29.607110 18570 webserver.cc:460] Webserver started at http://127.0.98.1:44379/ using document root <none> and password file <none>
I20260925 16:31:29.608153 18570 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:29.608289 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:29.608546 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:29.611253 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "49eb8dbaacde4298a8c1b108cd21f1d8"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "c3009b6c5cdefd89554625c0f5883650"
server_key_iv: "5a455c3cf8697170e07df8127249315b"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.611980 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "49eb8dbaacde4298a8c1b108cd21f1d8"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "c3009b6c5cdefd89554625c0f5883650"
server_key_iv: "5a455c3cf8697170e07df8127249315b"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.618090 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 16:31:29.622087 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.624109 18570 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:29.624315 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "49eb8dbaacde4298a8c1b108cd21f1d8"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "c3009b6c5cdefd89554625c0f5883650"
server_key_iv: "5a455c3cf8697170e07df8127249315b"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.624462 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:29.639885 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:29.644724 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:29.644968 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:29.645699 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:29.646791 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:29.646848 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.646908 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:29.646937 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.658959 18570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42163
I20260925 16:31:29.659086 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42163 every 8 connection(s)
I20260925 16:31:29.660166 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:29.668679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18570
I20260925 16:31:29.668866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:31:29.669335 392 external_mini_cluster.cc:1428] Setting key e92ab14676f4d7a37f6c0feadfa21c7a
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:29.681083 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:29.662989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55961 (local address 127.0.98.62:46881)
0925 16:31:29.663190 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:29.663196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:29.664869 (+ 1673us) server_negotiation.cc:407] Connection header received
0925 16:31:29.666107 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:29.666112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:29.666196 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:29.666325 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:29.668010 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.669116 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.671137 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.671456 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.671601 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:29.675878 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:29.675884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:29.675888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:29.675940 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:29.678076 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:29.678942 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:29.678946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:29.678948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:29.679078 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:29.679539 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:29.679543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:29.679545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:29.679876 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:29.679997 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:29.680663 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 16:31:29.680859 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:31:29.681982 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:29.663545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46881 (local address 127.0.98.1:55961)
0925 16:31:29.664598 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:29.664680 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:29.665705 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:29.666627 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:29.666636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:29.667106 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:29.667701 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:29.667719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.669448 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.669454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:29.670944 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:29.670969 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.671787 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.671795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:29.672193 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:29.673072 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:29.673096 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:29.675517 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:29.678319 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:29.678327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:29.678346 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:29.678774 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:29.679194 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:29.679197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:29.679198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:29.679415 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:29.680029 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:29.680037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:29.680361 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 16:31:29.680550 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 16:31:29.680887 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":236,"threads_started":1}
I20260925 16:31:29.683837 18699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46881
I20260925 16:31:29.684206 18699 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:31:29.685104 18699 heartbeater.cc:511] Master 127.0.98.62:46881 requested a full tablet report, sending...
I20260925 16:31:29.687849 18504 ts_manager.cc:194] Registered new tserver with Master: 49eb8dbaacde4298a8c1b108cd21f1d8 (127.0.98.1:42163)
I20260925 16:31:29.690006 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:31:29.706580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46881
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:29.850596 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:29.851054 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:29.851130 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:29.856987 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:29.857113 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:29.857209 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:29.864585 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:29.866353 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:29.868038 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:29.880474 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:29.880493 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:29.880468 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:29.880920 18704 server_base.cc:1034] running on GCE node
I20260925 16:31:29.881600 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:29.882586 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:29.883862 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790353889883809 us; error 75 us; skew 500 ppm
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:29.887876 18704 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:31:29.889268 18704 webserver.cc:460] Webserver started at http://127.0.98.2:37891/ using document root <none> and password file <none>
I20260925 16:31:29.890157 18704 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:29.890252 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:29.890524 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:29.893391 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4830d368ce74e00a92a9a5e6b52e2d0"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "72b3477053d9a3bb6e99ca744ef46404"
server_key_iv: "a2336b023d7c8168e0ddd4d0e222a8ec"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.894142 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4830d368ce74e00a92a9a5e6b52e2d0"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "72b3477053d9a3bb6e99ca744ef46404"
server_key_iv: "a2336b023d7c8168e0ddd4d0e222a8ec"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.899768 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 16:31:29.903774 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.905905 18704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:29.906109 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "e4830d368ce74e00a92a9a5e6b52e2d0"
format_stamp: "Formatted at 2026-09-25 16:31:29 on dist-test-slave-31kt"
server_key: "72b3477053d9a3bb6e99ca744ef46404"
server_key_iv: "a2336b023d7c8168e0ddd4d0e222a8ec"
server_key_version: "encryptionkey@0"
I20260925 16:31:29.906278 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:29.923354 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:29.927786 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:29.928033 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:29.928907 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:29.930117 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:29.930173 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.930238 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:29.930261 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:29.942898 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42869
I20260925 16:31:29.942940 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42869 every 8 connection(s)
I20260925 16:31:29.944097 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:29.946027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18704
I20260925 16:31:29.946146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:31:29.946455 392 external_mini_cluster.cc:1428] Setting key 58996d5a79f3899144b3e05e64de4e2e
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 16:31:29 dist-test-slave-31kt krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:29.968875 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:29.946755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40483 (local address 127.0.98.62:46881)
0925 16:31:29.946958 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:29.946962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:29.948509 (+ 1547us) server_negotiation.cc:407] Connection header received
0925 16:31:29.950647 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:29.950653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:29.950758 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:29.950967 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:29.952577 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.953461 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.955558 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.955971 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.956143 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:29.963533 (+ 7390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:29.963538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:29.963541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:29.963585 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:29.965514 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:29.966496 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:29.966500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:29.966502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:29.966636 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:29.967153 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:29.967156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:29.967158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:29.967461 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:29.967595 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:29.968473 (+ 878us) server_negotiation.cc:299] Negotiation successful
0925 16:31:29.968661 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:31:29.970042 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:29.947391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46881 (local address 127.0.98.2:40483)
0925 16:31:29.948239 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:29.948341 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:29.950229 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:29.951249 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:29.951259 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:29.951714 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:29.952333 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:29.952350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.953697 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.953701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:29.955308 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:29.955337 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:29.956367 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:29.956373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:29.956641 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:29.957707 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:29.957734 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:29.963119 (+ 5385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:29.965846 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:29.965856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:29.965879 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:29.966328 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:29.966763 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:29.966767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:29.966769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:29.967016 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:29.967645 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:29.967656 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:29.968074 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 16:31:29.968348 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 16:31:29.968768 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":195,"threads_started":1}
I20260925 16:31:29.971776 18833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46881
I20260925 16:31:29.972177 18833 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:29.972910 18833 heartbeater.cc:511] Master 127.0.98.62:46881 requested a full tablet report, sending...
I20260925 16:31:29.974496 18504 ts_manager.cc:194] Registered new tserver with Master: e4830d368ce74e00a92a9a5e6b52e2d0 (127.0.98.2:42869)
I20260925 16:31:29.975332 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:31:29.981417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46881
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:30.118881 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:30.119237 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:30.119315 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:30.124574 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:30.124680 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:30.124773 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:30.131886 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:30.133173 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:30.135149 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:31:30.149473 18838 server_base.cc:1034] running on GCE node
W20260925 16:31:30.149381 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:30.149561 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:30.149411 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:30.150391 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:30.151185 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:30.152434 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790353890152396 us; error 64 us; skew 500 ppm
Sep 25 16:31:30 dist-test-slave-31kt krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:30.156893 18838 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:31:30.158241 18838 webserver.cc:460] Webserver started at http://127.0.98.3:36519/ using document root <none> and password file <none>
I20260925 16:31:30.158948 18838 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:30.159029 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:30.159246 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:30.161837 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "767abf881ad74b089e4b57141f91308d"
format_stamp: "Formatted at 2026-09-25 16:31:30 on dist-test-slave-31kt"
server_key: "1f1756cf2fef596426784cb914907366"
server_key_iv: "cbe0eadb4f5cb4c624d4bd2b5b9873b0"
server_key_version: "encryptionkey@0"
I20260925 16:31:30.162555 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "767abf881ad74b089e4b57141f91308d"
format_stamp: "Formatted at 2026-09-25 16:31:30 on dist-test-slave-31kt"
server_key: "1f1756cf2fef596426784cb914907366"
server_key_iv: "cbe0eadb4f5cb4c624d4bd2b5b9873b0"
server_key_version: "encryptionkey@0"
I20260925 16:31:30.167332 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 16:31:30.170630 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:30.172343 18838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:31:30.172504 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "767abf881ad74b089e4b57141f91308d"
format_stamp: "Formatted at 2026-09-25 16:31:30 on dist-test-slave-31kt"
server_key: "1f1756cf2fef596426784cb914907366"
server_key_iv: "cbe0eadb4f5cb4c624d4bd2b5b9873b0"
server_key_version: "encryptionkey@0"
I20260925 16:31:30.172643 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:30.186758 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:30.191466 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:30.191741 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:30.192462 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:30.193634 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:30.193698 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:30.193789 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:30.193827 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 16:31:30.205454 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37821
I20260925 16:31:30.205535 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37821 every 8 connection(s)
I20260925 16:31:30.206632 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:30.210991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 18838
I20260925 16:31:30.211179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:31:30.211611 392 external_mini_cluster.cc:1428] Setting key 353d7ce505c5734e0c5266933eba594c
Sep 25 16:31:30 dist-test-slave-31kt krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:30.227386 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.209470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36323 (local address 127.0.98.62:46881)
0925 16:31:30.209708 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.209712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.211149 (+ 1437us) server_negotiation.cc:407] Connection header received
0925 16:31:30.213262 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.213266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.213339 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.213569 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:30.215578 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.216150 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.217522 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.217778 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.217888 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.221785 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:30.221791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:30.221794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:30.221837 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:30.223542 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:30.224426 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:30.224430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:30.224432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:30.224550 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:30.225247 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:30.225254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:30.225256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:30.225729 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:30.225940 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.226958 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.227177 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 16:31:30.228560 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.210042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46881 (local address 127.0.98.3:36323)
0925 16:31:30.210905 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.210972 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:30.213001 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:30.214090 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:30.214102 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:30.214695 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:30.215311 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.215329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.216321 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.216325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.217347 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.217361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.217962 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.217966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.218181 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.219197 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:30.219222 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:30.221435 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:30.223761 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:30.223771 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:30.223799 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:30.224291 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:30.224648 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:30.224652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:30.224654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:30.224954 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:30.226068 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:30.226080 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:30.226569 (+ 489us) client_negotiation.cc:769] Sending connection context
0925 16:31:30.226852 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 16:31:30.227327 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":203,"threads_started":1}
I20260925 16:31:30.230569 18967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46881
I20260925 16:31:30.230999 18967 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:30.231972 18967 heartbeater.cc:511] Master 127.0.98.62:46881 requested a full tablet report, sending...
I20260925 16:31:30.233896 18504 ts_manager.cc:194] Registered new tserver with Master: 767abf881ad74b089e4b57141f91308d (127.0.98.3:37821)
I20260925 16:31:30.234721 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36323
I20260925 16:31:30.241366 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:31:30.254397 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.243938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.0.98.62:46881)
0925 16:31:30.244191 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.244197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.244426 (+ 229us) server_negotiation.cc:407] Connection header received
0925 16:31:30.244590 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.244593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.244648 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.244722 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:30.245505 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.246115 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.247027 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.247260 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.247354 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.249219 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:30.249223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:30.249225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:30.249256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:30.250925 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:30.252116 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:30.252121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:30.252123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:30.252399 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:30.253157 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:30.253161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:30.253163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:30.253502 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:30.253681 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.254059 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.254200 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 16:31:30.259186 18504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:31:30.262795 18504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:31:30.281955 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.275022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42163 (local address 127.0.0.1:48572)
0925 16:31:30.275898 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.275944 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:30.276137 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:30.276928 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:30.276937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:30.276962 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:30.277144 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.277150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.278610 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.278613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.279842 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.279854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.281213 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.281216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.281345 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.281390 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 16:31:30.281529 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 16:31:30.281698 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":51,"thread_start_us":171,"threads_started":1}
I20260925 16:31:30.282044 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.275316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.98.1:42163)
0925 16:31:30.276072 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.276080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.276122 (+ 42us) server_negotiation.cc:407] Connection header received
0925 16:31:30.276402 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.276410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.276698 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.276859 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:30.277306 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.278486 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.280022 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.280872 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.280950 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.281109 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.281646 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.281858 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":148,"threads_started":1}
I20260925 16:31:30.282899 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.276016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37821 (local address 127.0.0.1:56534)
0925 16:31:30.277531 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.277544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:30.277655 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:30.278472 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:30.278475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:30.278496 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:30.278661 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.278668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.280035 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.280039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.281285 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.281293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.282430 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.282434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.282525 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.282563 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 16:31:30.282640 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 16:31:30.282734 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"mutex_wait_us":9,"thread_start_us":122,"threads_started":1}
I20260925 16:31:30.283253 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.275485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.2:42869)
0925 16:31:30.276327 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.276339 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.276832 (+ 493us) server_negotiation.cc:407] Connection header received
0925 16:31:30.277026 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.277031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.277275 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.277423 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:30.277987 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.279223 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.281204 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.282171 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.282290 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.282464 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.282911 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.283087 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":174,"threads_started":1}
I20260925 16:31:30.283411 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.276228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.0.98.3:37821)
0925 16:31:30.277388 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.277398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.277663 (+ 265us) server_negotiation.cc:407] Connection header received
0925 16:31:30.277820 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.277827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.278058 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.278217 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:30.278822 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.279864 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.281410 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.282302 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.282410 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.282575 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.283164 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.283267 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":172,"threads_started":1}
I20260925 16:31:30.283731 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.275056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.0.1:48348)
0925 16:31:30.276720 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.276734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:30.276870 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:30.277651 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:30.277653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:30.277672 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:30.277808 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.277815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.279428 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.279432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.281025 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.281038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.282641 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.282644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.282721 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.282739 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 16:31:30.283536 (+ 797us) client_negotiation.cc:240] Negotiation successful
0925 16:31:30.283631 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"mutex_wait_us":42,"thread_start_us":112,"threads_started":1}
I20260925 16:31:30.286824 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed0b3259b584b2a8e0e07f33c57f440 (DEFAULT_TABLE table=test-table [id=c1339ec9ef644a02975d99d603f1debc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:30.286671 18633 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed0b3259b584b2a8e0e07f33c57f440 (DEFAULT_TABLE table=test-table [id=c1339ec9ef644a02975d99d603f1debc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:30.286834 18901 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed0b3259b584b2a8e0e07f33c57f440 (DEFAULT_TABLE table=test-table [id=c1339ec9ef644a02975d99d603f1debc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:30.288317 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed0b3259b584b2a8e0e07f33c57f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:30.288317 18633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed0b3259b584b2a8e0e07f33c57f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:30.288479 18901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed0b3259b584b2a8e0e07f33c57f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:30.296589 18984 tablet_bootstrap.cc:492] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Bootstrap starting.
I20260925 16:31:30.298673 18985 tablet_bootstrap.cc:492] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: Bootstrap starting.
I20260925 16:31:30.299165 18984 tablet_bootstrap.cc:654] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:30.300488 18984 log.cc:824] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:30.301528 18986 tablet_bootstrap.cc:492] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: Bootstrap starting.
I20260925 16:31:30.301580 18985 tablet_bootstrap.cc:654] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:30.302652 18985 log.cc:824] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:30.304314 18986 tablet_bootstrap.cc:654] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:30.305263 18984 tablet_bootstrap.cc:492] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: No bootstrap required, opened a new log
I20260925 16:31:30.305263 18985 tablet_bootstrap.cc:492] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: No bootstrap required, opened a new log
I20260925 16:31:30.305569 18985 ts_tablet_manager.cc:1397] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:31:30.305569 18984 ts_tablet_manager.cc:1397] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 16:31:30.305841 18986 log.cc:824] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:30.308492 18986 tablet_bootstrap.cc:492] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: No bootstrap required, opened a new log
I20260925 16:31:30.308766 18986 ts_tablet_manager.cc:1397] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 16:31:30.310725 18984 raft_consensus.cc:348] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.311359 18984 raft_consensus.cc:374] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:30.311452 18984 raft_consensus.cc:729] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767abf881ad74b089e4b57141f91308d, State: Initialized, Role: FOLLOWER
I20260925 16:31:30.312276 18984 consensus_queue.cc:260] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.313247 18985 raft_consensus.cc:348] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.313246 18986 raft_consensus.cc:348] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.313694 18985 raft_consensus.cc:374] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:30.313614 18986 raft_consensus.cc:374] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:30.313870 18986 raft_consensus.cc:729] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4830d368ce74e00a92a9a5e6b52e2d0, State: Initialized, Role: FOLLOWER
I20260925 16:31:30.313890 18985 raft_consensus.cc:729] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49eb8dbaacde4298a8c1b108cd21f1d8, State: Initialized, Role: FOLLOWER
I20260925 16:31:30.314073 18984 ts_tablet_manager.cc:1428] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260925 16:31:30.314599 18967 heartbeater.cc:503] Master 127.0.98.62:46881 was elected leader, sending a full tablet report...
I20260925 16:31:30.314584 18985 consensus_queue.cc:260] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.314839 18986 consensus_queue.cc:260] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.316120 18985 ts_tablet_manager.cc:1428] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260925 16:31:30.316654 18699 heartbeater.cc:503] Master 127.0.98.62:46881 was elected leader, sending a full tablet report...
I20260925 16:31:30.317142 18986 ts_tablet_manager.cc:1428] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260925 16:31:30.317627 18833 heartbeater.cc:503] Master 127.0.98.62:46881 was elected leader, sending a full tablet report...
I20260925 16:31:30.408408 18991 raft_consensus.cc:482] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:30.408726 18991 raft_consensus.cc:504] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.410356 18991 leader_election.cc:288] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4830d368ce74e00a92a9a5e6b52e2d0 (127.0.98.2:42869), 767abf881ad74b089e4b57141f91308d (127.0.98.3:37821)
W20260925 16:31:30.411885 18700 tablet.cc:2369] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:31:30.415535 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.410915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60805 (local address 127.0.98.2:42869)
0925 16:31:30.411071 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.411075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.411317 (+ 242us) server_negotiation.cc:407] Connection header received
0925 16:31:30.411788 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.411793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.411864 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.411976 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:30.412636 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.413212 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.414467 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.414931 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.415029 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.415110 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.415298 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.415386 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:31:30.415561 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.410837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.98.1:60805)
0925 16:31:30.411206 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.411223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:30.411477 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:30.412021 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:30.412026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:30.412051 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:30.412231 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.412240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.413333 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.413337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.414334 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.414345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.415059 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.415062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.415158 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.415203 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 16:31:30.415269 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 16:31:30.415363 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":42,"thread_start_us":138,"threads_started":1}
I20260925 16:31:30.415593 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.411512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37821 (local address 127.0.98.1:45819)
0925 16:31:30.412081 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.412096 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:30.412243 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:30.412555 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:30.412559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:30.412581 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:30.412699 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.412704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.413593 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.413597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.414474 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:30.414483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.415160 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.415164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:30.415282 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.415301 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:31:30.415365 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:31:30.415434 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":22,"thread_start_us":176,"threads_started":1}
I20260925 16:31:30.415601 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.411535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45819 (local address 127.0.98.3:37821)
0925 16:31:30.411673 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.411678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.412180 (+ 502us) server_negotiation.cc:407] Connection header received
0925 16:31:30.412353 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.412357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.412419 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.412502 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:30.412897 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.413472 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.414602 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.415050 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.415145 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.415215 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.415396 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.415482 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:31:30.416495 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed0b3259b584b2a8e0e07f33c57f440" candidate_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" is_pre_election: true
I20260925 16:31:30.416495 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed0b3259b584b2a8e0e07f33c57f440" candidate_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767abf881ad74b089e4b57141f91308d" is_pre_election: true
I20260925 16:31:30.416926 18787 raft_consensus.cc:2454] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49eb8dbaacde4298a8c1b108cd21f1d8 in term 0.
I20260925 16:31:30.416927 18921 raft_consensus.cc:2454] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49eb8dbaacde4298a8c1b108cd21f1d8 in term 0.
I20260925 16:31:30.417825 18586 leader_election.cc:302] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49eb8dbaacde4298a8c1b108cd21f1d8, e4830d368ce74e00a92a9a5e6b52e2d0; no voters:
I20260925 16:31:30.418263 18991 raft_consensus.cc:2781] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:31:30.418378 18991 raft_consensus.cc:482] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:31:30.418431 18991 raft_consensus.cc:3037] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:30.419688 18991 raft_consensus.cc:504] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.420142 18991 leader_election.cc:288] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [CANDIDATE]: Term 1 election: Requested vote from peers e4830d368ce74e00a92a9a5e6b52e2d0 (127.0.98.2:42869), 767abf881ad74b089e4b57141f91308d (127.0.98.3:37821)
I20260925 16:31:30.420889 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed0b3259b584b2a8e0e07f33c57f440" candidate_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0"
I20260925 16:31:30.420889 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed0b3259b584b2a8e0e07f33c57f440" candidate_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767abf881ad74b089e4b57141f91308d"
I20260925 16:31:30.421123 18787 raft_consensus.cc:3037] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:30.421123 18921 raft_consensus.cc:3037] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:30.422721 18787 raft_consensus.cc:2454] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49eb8dbaacde4298a8c1b108cd21f1d8 in term 1.
I20260925 16:31:30.422721 18921 raft_consensus.cc:2454] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49eb8dbaacde4298a8c1b108cd21f1d8 in term 1.
I20260925 16:31:30.423311 18586 leader_election.cc:302] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49eb8dbaacde4298a8c1b108cd21f1d8, 767abf881ad74b089e4b57141f91308d; no voters:
I20260925 16:31:30.423584 18991 raft_consensus.cc:2781] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:30.423898 18991 raft_consensus.cc:686] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 1 LEADER]: Becoming Leader. State: Replica: 49eb8dbaacde4298a8c1b108cd21f1d8, State: Running, Role: LEADER
I20260925 16:31:30.424683 18991 consensus_queue.cc:237] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } }
I20260925 16:31:30.429189 18505 catalog_manager.cc:5495] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49eb8dbaacde4298a8c1b108cd21f1d8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49eb8dbaacde4298a8c1b108cd21f1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 } health_report { overall_health: UNKNOWN } } }
I20260925 16:31:30.445382 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.440818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.0.98.1:42163)
0925 16:31:30.440975 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.440980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.440999 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:31:30.441170 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.441173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.441257 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.441487 (+ 230us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:31:30.441798 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.442433 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.443373 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.443603 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.443699 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.444339 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:31:30.444842 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:31:30.444999 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.445139 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.445199 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 16:31:30.445699 18834 tablet.cc:2369] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:31:30.454077 18921 raft_consensus.cc:1261] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 1 FOLLOWER]: Refusing update from remote peer 49eb8dbaacde4298a8c1b108cd21f1d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:30.454077 18787 raft_consensus.cc:1261] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 1 FOLLOWER]: Refusing update from remote peer 49eb8dbaacde4298a8c1b108cd21f1d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:30.455410 18995 consensus_queue.cc:1035] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767abf881ad74b089e4b57141f91308d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:30.455827 18991 consensus_queue.cc:1035] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4830d368ce74e00a92a9a5e6b52e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 16:31:30.457964 18968 tablet.cc:2369] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:31:30.466713 18998 mvcc.cc:205] Tried to move back new op lower bound from 7333289535287328768 to 7333289535182778368. Current Snapshot: MvccSnapshot[applied={T|T < 7333289535287328768}]
I20260925 16:31:30.480381 18505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:30.480639 18505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:30.484089 18505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd0fb67e1b9a437293b9baf1d6f7d3a1: Sending DeleteTablet for 3 replicas of tablet 6ed0b3259b584b2a8e0e07f33c57f440
I20260925 16:31:30.485337 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed0b3259b584b2a8e0e07f33c57f440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48348
I20260925 16:31:30.485337 18633 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed0b3259b584b2a8e0e07f33c57f440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48572
I20260925 16:31:30.485667 18901 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed0b3259b584b2a8e0e07f33c57f440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56534
I20260925 16:31:30.486231 19008 tablet_replica.cc:330] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8: stopping tablet replica
I20260925 16:31:30.486802 19009 tablet_replica.cc:330] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: stopping tablet replica
I20260925 16:31:30.486888 19008 raft_consensus.cc:2229] T 6ed0b3259b584b2a8e0e07f33c57f440 P 49eb8dbaacde4298a8c1b108cd21f1d8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:31:30.487020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18570
I20260925 16:31:30.488560 19009 raft_consensus.cc:2229] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:30.488968 19009 raft_consensus.cc:2258] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:30.490892 19009 ts_tablet_manager.cc:1905] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:31:30.493875 18494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42163 (error 108)
I20260925 16:31:30.493964 19009 ts_tablet_manager.cc:1918] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:30.494096 19009 log.cc:1196] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/6ed0b3259b584b2a8e0e07f33c57f440
I20260925 16:31:30.494447 19009 ts_tablet_manager.cc:1939] T 6ed0b3259b584b2a8e0e07f33c57f440 P 767abf881ad74b089e4b57141f91308d: Deleting consensus metadata
I20260925 16:31:30.494438 19007 tablet_replica.cc:330] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0: stopping tablet replica
I20260925 16:31:30.494871 19007 raft_consensus.cc:2229] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:30.494956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18704
I20260925 16:31:30.495337 19007 raft_consensus.cc:2258] T 6ed0b3259b584b2a8e0e07f33c57f440 P e4830d368ce74e00a92a9a5e6b52e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:30.496021 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.494914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42163 (local address 127.0.0.1:48600)
0925 16:31:30.495774 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.495893 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":722}
I20260925 16:31:30.496441 18492 catalog_manager.cc:4841] TS 767abf881ad74b089e4b57141f91308d (127.0.98.3:37821): tablet 6ed0b3259b584b2a8e0e07f33c57f440 (table test-table [id=c1339ec9ef644a02975d99d603f1debc]) successfully deleted
W20260925 16:31:30.496520 18494 catalog_manager.cc:4584] TS 49eb8dbaacde4298a8c1b108cd21f1d8 (127.0.98.1:42163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ed0b3259b584b2a8e0e07f33c57f440: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42163: connect: Connection refused (error 111)
I20260925 16:31:30.501942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18838
I20260925 16:31:30.504403 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.503890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42869 (local address 127.0.0.1:48362)
0925 16:31:30.504129 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:30.504270 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 16:31:30.508213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 18476
2026-09-25T16:31:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2564 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T16:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:30Z Disabled control of system clock
I20260925 16:31:30.543602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34149
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:31:30.692667 19014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:30.693095 19014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:30.693140 19014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:30.698195 19014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:30.698320 19014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:30.698338 19014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:30.698354 19014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:30.698374 19014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:30.704216 19014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:30.705884 19014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:30.707844 19014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:30.718735 19020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:30.718735 19019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:30.718735 19022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:30.719309 19014 server_base.cc:1034] running on GCE node
I20260925 16:31:30.720320 19014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:30.722066 19014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:30.723348 19014 hybrid_clock.cc:648] HybridClock initialized: now 1790353890723294 us; error 58 us; skew 500 ppm
I20260925 16:31:30.726275 19014 webserver.cc:460] Webserver started at http://127.0.98.62:45199/ using document root <none> and password file <none>
I20260925 16:31:30.727059 19014 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:30.727149 19014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:30.727408 19014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:30.730036 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "8cc0408db4194d56bf0a8fda13ce7e53"
format_stamp: "Formatted at 2026-09-25 16:31:30 on dist-test-slave-31kt"
server_key: "04ca8601047744d866c7dc2fbe55db5c"
server_key_iv: "07406e30524cb7b21f6ca2a57e361a5c"
server_key_version: "encryptionkey@0"
I20260925 16:31:30.730719 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cc0408db4194d56bf0a8fda13ce7e53"
format_stamp: "Formatted at 2026-09-25 16:31:30 on dist-test-slave-31kt"
server_key: "04ca8601047744d866c7dc2fbe55db5c"
server_key_iv: "07406e30524cb7b21f6ca2a57e361a5c"
server_key_version: "encryptionkey@0"
I20260925 16:31:30.735526 19014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:31:30.739009 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:30.741055 19014 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:30.741245 19014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "8cc0408db4194d56bf0a8fda13ce7e53"
format_stamp: "Formatted at 2026-09-25 16:31:30 on dist-test-slave-31kt"
server_key: "04ca8601047744d866c7dc2fbe55db5c"
server_key_iv: "07406e30524cb7b21f6ca2a57e361a5c"
server_key_version: "encryptionkey@0"
I20260925 16:31:30.741432 19014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:30.768002 19014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:30.768860 19014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:30.769068 19014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:30.777989 19014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34149
I20260925 16:31:30.778044 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34149 every 8 connection(s)
I20260925 16:31:30.779790 19014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:30.783069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19014
I20260925 16:31:30.783246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:30.783610 392 external_mini_cluster.cc:1428] Setting key 2ee0ac2b2e5d6ef24cedf605947ff176
I20260925 16:31:30.785966 19081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:30.794031 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: Bootstrap starting.
I20260925 16:31:30.796840 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:30.785569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.0.98.62:34149)
0925 16:31:30.786528 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:30.786550 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:30.786605 (+ 55us) server_negotiation.cc:407] Connection header received
0925 16:31:30.787505 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:30.787530 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:30.788114 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:30.788544 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:30.789193 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.790536 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.791996 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:30.792667 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:30.792864 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:30.794150 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:30.794158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:30.794179 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:30.794864 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:30.794958 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:30.794989 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:30.795100 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:30.795391 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 16:31:30.795777 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":235,"threads_started":1}
I20260925 16:31:30.798074 19081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:30.799114 19081 log.cc:824] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:30.801921 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: No bootstrap required, opened a new log
I20260925 16:31:30.806084 19081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } }
I20260925 16:31:30.806411 19081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:30.806463 19081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc0408db4194d56bf0a8fda13ce7e53, State: Initialized, Role: FOLLOWER
I20260925 16:31:30.806969 19081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } }
I20260925 16:31:30.807116 19081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:30.807194 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:30.807300 19081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:30.808353 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } }
I20260925 16:31:30.808882 19081 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc0408db4194d56bf0a8fda13ce7e53; no voters:
I20260925 16:31:30.809288 19081 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:30.809520 19086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:30.809953 19086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc0408db4194d56bf0a8fda13ce7e53, State: Running, Role: LEADER
I20260925 16:31:30.810444 19086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } }
I20260925 16:31:30.810802 19081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:30.812865 19087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } }
I20260925 16:31:30.812821 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc0408db4194d56bf0a8fda13ce7e53. Latest consensus state: current_term: 1 leader_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc0408db4194d56bf0a8fda13ce7e53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34149 } } }
I20260925 16:31:30.813066 19087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:30.813093 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53 [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:30.816670 19101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:30.816789 19101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:30.816864 19100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:30.817708 19100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:30.825202 19100 catalog_manager.cc:1347] Generated new cluster ID: 4abac934d8e74bbf91626fa927a6dbe0
I20260925 16:31:30.825301 19100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:30.840253 19100 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:30.842170 19100 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:30.851263 19100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: Generated new TSK 0
I20260925 16:31:30.852535 19100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:30.857626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34149
--builtin_ntp_servers=127.0.98.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:31:30.995764 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:30.996121 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:30.996179 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:31.001212 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:31.001330 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:31.001420 19105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:31.006927 19105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:31.008759 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:31.011441 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:31.025816 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:31.025846 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:31.025846 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:31.026233 19105 server_base.cc:1034] running on GCE node
I20260925 16:31:31.027056 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:31.028091 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:31.029316 19105 hybrid_clock.cc:648] HybridClock initialized: now 1790353891029275 us; error 66 us; skew 500 ppm
I20260925 16:31:31.032205 19105 webserver.cc:460] Webserver started at http://127.0.98.1:45229/ using document root <none> and password file <none>
I20260925 16:31:31.033093 19105 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:31.033198 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:31.033465 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:31.036437 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "dddd205ff482a4280a3de353ceb82bde"
server_key_iv: "d967239813e3fd8c1ced93eb4432a929"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.037254 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "dddd205ff482a4280a3de353ceb82bde"
server_key_iv: "d967239813e3fd8c1ced93eb4432a929"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.042631 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 16:31:31.046773 19119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.048986 19105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:31.049242 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "dddd205ff482a4280a3de353ceb82bde"
server_key_iv: "d967239813e3fd8c1ced93eb4432a929"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.049432 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:31.069667 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:31.070741 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:31.070968 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:31.071774 19105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:31.073060 19105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:31.073114 19105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.073179 19105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:31.073204 19105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.085049 19105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43707
I20260925 16:31:31.085139 19232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43707 every 8 connection(s)
I20260925 16:31:31.086539 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:31.087905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19105
I20260925 16:31:31.088058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:31:31.088379 392 external_mini_cluster.cc:1428] Setting key f7f70a75dea88e022017c979e49201f4
I20260925 16:31:31.091168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34149
--builtin_ntp_servers=127.0.98.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:31:31.101930 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.089280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33591 (local address 127.0.98.62:34149)
0925 16:31:31.089420 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.089424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.091759 (+ 2335us) server_negotiation.cc:407] Connection header received
0925 16:31:31.093715 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.093721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.093862 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.094024 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:31.095715 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.096719 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.098531 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.098994 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.099157 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.101084 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:31.101090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:31.101094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:31.101119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:31.101156 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:31.101171 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:31.101275 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.101532 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.101660 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:31:31.102944 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.090297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34149 (local address 127.0.98.1:33591)
0925 16:31:31.091520 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.091605 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.093372 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.094439 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.094499 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.094660 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:31.095461 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.095485 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.097071 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.097076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.098321 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.098344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.099278 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.099282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.099682 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.100610 (+ 928us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:31.100612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:31.100708 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:31.100815 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:31.101337 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:31.101346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:31.101352 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.101504 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.101957 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":193,"threads_started":1}
I20260925 16:31:31.104635 19233 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34149
I20260925 16:31:31.105072 19233 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:31.106034 19233 heartbeater.cc:511] Master 127.0.98.62:34149 requested a full tablet report, sending...
I20260925 16:31:31.108606 19040 ts_manager.cc:194] Registered new tserver with Master: f84ab829a32b40c2a90a9b8ca1f6d7dc (127.0.98.1:43707)
I20260925 16:31:31.110633 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33591
W20260925 16:31:31.235317 19236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:31.235724 19236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:31.235769 19236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:31.240571 19236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:31.240694 19236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:31.240763 19236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:31.246614 19236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:31.248224 19236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:31.250231 19236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:31.263159 19242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:31.263173 19244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:31.263159 19241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:31.263411 19236 server_base.cc:1034] running on GCE node
I20260925 16:31:31.264330 19236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:31.265399 19236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:31.266638 19236 hybrid_clock.cc:648] HybridClock initialized: now 1790353891266620 us; error 91 us; skew 500 ppm
I20260925 16:31:31.269028 19236 webserver.cc:460] Webserver started at http://127.0.98.2:36951/ using document root <none> and password file <none>
I20260925 16:31:31.269737 19236 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:31.269868 19236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:31.270102 19236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:31.272756 19236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "0eb3ee0d1fb542b1847358cf7e63eb54"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "c7e844c52b5ae43b75311ae7a462954d"
server_key_iv: "fa2257c41462c04b7b3cda1a5ebd34ee"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.273422 19236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0eb3ee0d1fb542b1847358cf7e63eb54"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "c7e844c52b5ae43b75311ae7a462954d"
server_key_iv: "fa2257c41462c04b7b3cda1a5ebd34ee"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.278463 19236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:31:31.282142 19250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.284071 19236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:31.284288 19236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "0eb3ee0d1fb542b1847358cf7e63eb54"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "c7e844c52b5ae43b75311ae7a462954d"
server_key_iv: "fa2257c41462c04b7b3cda1a5ebd34ee"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.284436 19236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:31.309535 19236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:31.310417 19236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:31.310626 19236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:31.311340 19236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:31.312597 19236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:31.312657 19236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.312726 19236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:31.312752 19236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.324947 19236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33889
I20260925 16:31:31.325022 19363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33889 every 8 connection(s)
I20260925 16:31:31.326141 19236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:31.330930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19236
I20260925 16:31:31.331048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:31:31.331315 392 external_mini_cluster.cc:1428] Setting key edc26eef0170ce115f1b30cd8e48bf67
I20260925 16:31:31.334158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34149
--builtin_ntp_servers=127.0.98.20:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:31:31.339991 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.328917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44527 (local address 127.0.98.62:34149)
0925 16:31:31.329112 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.329116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.330729 (+ 1613us) server_negotiation.cc:407] Connection header received
0925 16:31:31.332647 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.332650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.332723 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.332852 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:31.334325 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.335190 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.336988 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.337262 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.337381 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.339334 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:31.339339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:31.339341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:31.339361 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:31.339387 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:31.339397 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:31.339462 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.339665 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.339778 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 16:31:31.341275 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.329451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34149 (local address 127.0.98.2:44527)
0925 16:31:31.330485 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.330554 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.332287 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.333151 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.333164 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.333313 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:31.334102 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.334121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.335500 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.335505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.336737 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.336756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.337498 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.337503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.337832 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.338887 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:31.338889 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:31.338959 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:31.339079 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:31.339480 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:31.339497 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:31.339504 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.339650 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.340118 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":174,"threads_started":1}
I20260925 16:31:31.343349 19364 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34149
I20260925 16:31:31.343777 19364 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:31.344625 19364 heartbeater.cc:511] Master 127.0.98.62:34149 requested a full tablet report, sending...
I20260925 16:31:31.346408 19040 ts_manager.cc:194] Registered new tserver with Master: 0eb3ee0d1fb542b1847358cf7e63eb54 (127.0.98.2:33889)
I20260925 16:31:31.347234 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44527
W20260925 16:31:31.482901 19367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:31.483232 19367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:31.483287 19367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:31.487515 19367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:31.487603 19367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:31.487686 19367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:31.493083 19367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:31.494432 19367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:31.495993 19367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:31.507522 19375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:31.507551 19372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:31.507545 19373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:31.508071 19367 server_base.cc:1034] running on GCE node
I20260925 16:31:31.508620 19367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:31.509474 19367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:31.510684 19367 hybrid_clock.cc:648] HybridClock initialized: now 1790353891510683 us; error 61 us; skew 500 ppm
I20260925 16:31:31.513722 19367 webserver.cc:460] Webserver started at http://127.0.98.3:34511/ using document root <none> and password file <none>
I20260925 16:31:31.514636 19367 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:31.514737 19367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:31.515012 19367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:31.518000 19367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "4f3e315f08094d1298d762851de82ff2"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "d914cd0493673f54ceb4a5870aacf942"
server_key_iv: "7ec4efaa3a2250c7af8624bcceb7b6a9"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.518813 19367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4f3e315f08094d1298d762851de82ff2"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "d914cd0493673f54ceb4a5870aacf942"
server_key_iv: "7ec4efaa3a2250c7af8624bcceb7b6a9"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.524187 19367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:31.528069 19381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.530354 19367 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:31.530583 19367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "4f3e315f08094d1298d762851de82ff2"
format_stamp: "Formatted at 2026-09-25 16:31:31 on dist-test-slave-31kt"
server_key: "d914cd0493673f54ceb4a5870aacf942"
server_key_iv: "7ec4efaa3a2250c7af8624bcceb7b6a9"
server_key_version: "encryptionkey@0"
I20260925 16:31:31.530766 19367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:31.560828 19367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:31.561810 19367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:31.562006 19367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:31.562767 19367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:31.564070 19367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:31.564124 19367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.564188 19367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:31.564213 19367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:31.575613 19367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44507
I20260925 16:31:31.575652 19494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44507 every 8 connection(s)
I20260925 16:31:31.576890 19367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:31.584745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19367
I20260925 16:31:31.584911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:31:31.585278 392 external_mini_cluster.cc:1428] Setting key f33ee72eb94d157ee49e8fad2086d368
I20260925 16:31:31.590643 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.579683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53799 (local address 127.0.98.62:34149)
0925 16:31:31.579934 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.579941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.581678 (+ 1737us) server_negotiation.cc:407] Connection header received
0925 16:31:31.583442 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.583447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.583531 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.583648 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:31.585060 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.585911 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.587618 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.588057 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.588184 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.589939 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:31.589943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:31.589952 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:31.589972 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:31.590003 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:31.590013 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:31.590095 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.590321 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.590418 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 16:31:31.592343 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.580285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34149 (local address 127.0.98.3:53799)
0925 16:31:31.581441 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.581506 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.583026 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.583943 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.583954 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.584065 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:31.584698 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.584722 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.586132 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.586136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.587464 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.587481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.588319 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.588324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.588570 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.589473 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:31.589475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:31.589545 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:31.589663 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:31.590132 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:31.590142 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:31.590149 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.590297 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.591090 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":208,"threads_started":1}
I20260925 16:31:31.594436 19495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34149
I20260925 16:31:31.594828 19495 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:31.595583 19495 heartbeater.cc:511] Master 127.0.98.62:34149 requested a full tablet report, sending...
I20260925 16:31:31.597283 19040 ts_manager.cc:194] Registered new tserver with Master: 4f3e315f08094d1298d762851de82ff2 (127.0.98.3:44507)
I20260925 16:31:31.598093 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53799
I20260925 16:31:31.600692 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:31:31.608605 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.603750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.0.98.62:34149)
0925 16:31:31.603915 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.603920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.604048 (+ 128us) server_negotiation.cc:407] Connection header received
0925 16:31:31.604306 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.604311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.604381 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.604500 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:31.604943 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.605519 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.606637 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.606906 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.607015 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.608036 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:31.608039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:31.608042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:31.608058 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:31.608086 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:31.608094 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:31.608214 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.608394 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.608438 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 16:31:31.613179 19040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:31:31.617463 19040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:31:31.636286 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.629957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44507 (local address 127.0.0.1:48672)
0925 16:31:31.630880 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.630894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.630980 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.631773 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.631776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.631796 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:31.631904 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.631911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.633350 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.633354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.634550 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.634562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.635610 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.635613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.635751 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.635794 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.635922 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.636095 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":35,"thread_start_us":115,"threads_started":1}
I20260925 16:31:31.636413 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.630438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.0.98.3:44507)
0925 16:31:31.631246 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.631260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.631285 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:31:31.631370 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.631376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.631611 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.632048 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:31.632196 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.633252 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.634670 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.635436 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.635546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.635817 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.636028 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.636186 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":156,"threads_started":1}
I20260925 16:31:31.636658 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.629082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.98.1:43707)
0925 16:31:31.629600 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.629607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.630089 (+ 482us) server_negotiation.cc:407] Connection header received
0925 16:31:31.630387 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.630391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.630849 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.631006 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:31.631493 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.632819 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.635191 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.636032 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.636161 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.636344 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.636394 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.636501 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":107,"threads_started":1}
I20260925 16:31:31.637202 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.628933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43707 (local address 127.0.0.1:58492)
0925 16:31:31.629913 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.629981 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.630197 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.631201 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.631203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.631220 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:31.631312 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.631317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.632960 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.632963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.634973 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.634999 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.636175 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.636178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.636272 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.636294 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.636361 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.637058 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":134,"thread_start_us":150,"threads_started":1}
I20260925 16:31:31.638702 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.628989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33889 (local address 127.0.0.1:48952)
0925 16:31:31.629911 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.629981 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.630179 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.631072 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.631079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.631106 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:31.631279 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.631286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.632947 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.632951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.634973 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.634999 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.638206 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.638211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.638337 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.638368 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.638448 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.638549 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":230,"thread_start_us":137,"threads_started":1}
I20260925 16:31:31.639235 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.629295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.0.98.2:33889)
0925 16:31:31.630222 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.630234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.630259 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:31:31.630356 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.630363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.630849 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.631432 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:31.631779 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.632765 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.636882 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.637793 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.637918 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.638115 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.638944 (+ 829us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.639084 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":178,"threads_started":1}
I20260925 16:31:31.640492 19429 tablet_service.cc:1467] Processing CreateTablet for tablet 83d0290a1b154a93a3e04f8acc9c65d3 (DEFAULT_TABLE table=test-table [id=caa947afbd6d4109ba5c4bd286d5bafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:31.640492 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 83d0290a1b154a93a3e04f8acc9c65d3 (DEFAULT_TABLE table=test-table [id=caa947afbd6d4109ba5c4bd286d5bafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:31.641934 19429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d0290a1b154a93a3e04f8acc9c65d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:31.641934 19167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d0290a1b154a93a3e04f8acc9c65d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:31.645387 19298 tablet_service.cc:1467] Processing CreateTablet for tablet 83d0290a1b154a93a3e04f8acc9c65d3 (DEFAULT_TABLE table=test-table [id=caa947afbd6d4109ba5c4bd286d5bafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:31.647231 19298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d0290a1b154a93a3e04f8acc9c65d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:31.652196 19513 tablet_bootstrap.cc:492] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Bootstrap starting.
I20260925 16:31:31.654031 19512 tablet_bootstrap.cc:492] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: Bootstrap starting.
I20260925 16:31:31.655674 19513 tablet_bootstrap.cc:654] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:31.656893 19512 tablet_bootstrap.cc:654] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:31.656994 19513 log.cc:824] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:31.657917 19514 tablet_bootstrap.cc:492] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Bootstrap starting.
I20260925 16:31:31.659447 19512 log.cc:824] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:31.660048 19513 tablet_bootstrap.cc:492] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: No bootstrap required, opened a new log
I20260925 16:31:31.660363 19513 ts_tablet_manager.cc:1397] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 16:31:31.661926 19514 tablet_bootstrap.cc:654] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:31.662981 19512 tablet_bootstrap.cc:492] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: No bootstrap required, opened a new log
I20260925 16:31:31.663096 19514 log.cc:824] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:31.663249 19512 ts_tablet_manager.cc:1397] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 16:31:31.664927 19513 raft_consensus.cc:348] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.665313 19513 raft_consensus.cc:374] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:31.665366 19513 raft_consensus.cc:729] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3e315f08094d1298d762851de82ff2, State: Initialized, Role: FOLLOWER
I20260925 16:31:31.666168 19514 tablet_bootstrap.cc:492] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: No bootstrap required, opened a new log
I20260925 16:31:31.666118 19513 consensus_queue.cc:260] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.666433 19514 ts_tablet_manager.cc:1397] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 16:31:31.667636 19513 ts_tablet_manager.cc:1428] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:31:31.668138 19495 heartbeater.cc:503] Master 127.0.98.62:34149 was elected leader, sending a full tablet report...
I20260925 16:31:31.668109 19512 raft_consensus.cc:348] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.668460 19512 raft_consensus.cc:374] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:31.668545 19512 raft_consensus.cc:729] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84ab829a32b40c2a90a9b8ca1f6d7dc, State: Initialized, Role: FOLLOWER
I20260925 16:31:31.669095 19512 consensus_queue.cc:260] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.670037 19233 heartbeater.cc:503] Master 127.0.98.62:34149 was elected leader, sending a full tablet report...
I20260925 16:31:31.670457 19512 ts_tablet_manager.cc:1428] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:31:31.671821 19514 raft_consensus.cc:348] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.672125 19514 raft_consensus.cc:374] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:31.672189 19514 raft_consensus.cc:729] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb3ee0d1fb542b1847358cf7e63eb54, State: Initialized, Role: FOLLOWER
I20260925 16:31:31.672813 19514 consensus_queue.cc:260] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.674216 19514 ts_tablet_manager.cc:1428] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 16:31:31.674401 19364 heartbeater.cc:503] Master 127.0.98.62:34149 was elected leader, sending a full tablet report...
W20260925 16:31:31.827589 19365 tablet.cc:2369] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:31:31.828162 19496 tablet.cc:2369] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 16:31:31.838478 19234 tablet.cc:2369] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:31:31.906174 19519 raft_consensus.cc:482] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:31.906418 19519 raft_consensus.cc:504] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.907711 19519 leader_election.cc:288] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eb3ee0d1fb542b1847358cf7e63eb54 (127.0.98.2:33889), 4f3e315f08094d1298d762851de82ff2 (127.0.98.3:44507)
I20260925 16:31:31.914539 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.908552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43085 (local address 127.0.98.3:44507)
0925 16:31:31.908732 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.908737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.908754 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:31:31.908829 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.908832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.908887 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.908971 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:31.909449 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.910437 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.912768 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.913444 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.913595 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.913672 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.914225 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.914344 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 16:31:31.914539 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.908276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56905 (local address 127.0.98.2:33889)
0925 16:31:31.908454 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.908459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.908837 (+ 378us) server_negotiation.cc:407] Connection header received
0925 16:31:31.908983 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.908986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.909064 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.909138 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:31.909566 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.910436 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.912689 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.913430 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.913570 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.913673 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.914221 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.914344 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 16:31:31.914578 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.908111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44507 (local address 127.0.98.1:43085)
0925 16:31:31.908481 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.908502 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.908662 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.909016 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.909019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.909034 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:31.909175 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.909181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.910631 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.910636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.912550 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.912559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.913726 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.913737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.913995 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.914064 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.914170 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.914340 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":264,"thread_start_us":140,"threads_started":1}
I20260925 16:31:31.914587 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.908170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33889 (local address 127.0.98.1:56905)
0925 16:31:31.908704 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.908719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:31.908861 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:31.909169 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:31.909171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:31.909184 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:31.909403 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.909414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.910631 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.910636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.912438 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:31.912467 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.913726 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.913737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:31.913997 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.914086 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 16:31:31.914186 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 16:31:31.914340 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":183,"thread_start_us":161,"threads_started":1}
I20260925 16:31:31.915597 19449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d0290a1b154a93a3e04f8acc9c65d3" candidate_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3e315f08094d1298d762851de82ff2" is_pre_election: true
I20260925 16:31:31.915818 19318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d0290a1b154a93a3e04f8acc9c65d3" candidate_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" is_pre_election: true
I20260925 16:31:31.916165 19449 raft_consensus.cc:2454] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84ab829a32b40c2a90a9b8ca1f6d7dc in term 0.
I20260925 16:31:31.916164 19318 raft_consensus.cc:2454] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84ab829a32b40c2a90a9b8ca1f6d7dc in term 0.
I20260925 16:31:31.916868 19122 leader_election.cc:302] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb3ee0d1fb542b1847358cf7e63eb54, f84ab829a32b40c2a90a9b8ca1f6d7dc; no voters:
I20260925 16:31:31.917223 19519 raft_consensus.cc:2781] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:31:31.917321 19519 raft_consensus.cc:482] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:31:31.917367 19519 raft_consensus.cc:3037] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:31.918790 19519 raft_consensus.cc:504] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.919248 19519 leader_election.cc:288] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [CANDIDATE]: Term 1 election: Requested vote from peers 0eb3ee0d1fb542b1847358cf7e63eb54 (127.0.98.2:33889), 4f3e315f08094d1298d762851de82ff2 (127.0.98.3:44507)
I20260925 16:31:31.919999 19318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d0290a1b154a93a3e04f8acc9c65d3" candidate_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54"
I20260925 16:31:31.920040 19449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d0290a1b154a93a3e04f8acc9c65d3" candidate_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3e315f08094d1298d762851de82ff2"
I20260925 16:31:31.920189 19449 raft_consensus.cc:3037] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:31.920189 19318 raft_consensus.cc:3037] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:31.922248 19449 raft_consensus.cc:2454] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84ab829a32b40c2a90a9b8ca1f6d7dc in term 1.
I20260925 16:31:31.922242 19318 raft_consensus.cc:2454] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84ab829a32b40c2a90a9b8ca1f6d7dc in term 1.
I20260925 16:31:31.922947 19122 leader_election.cc:302] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0eb3ee0d1fb542b1847358cf7e63eb54, 4f3e315f08094d1298d762851de82ff2, f84ab829a32b40c2a90a9b8ca1f6d7dc; no voters:
I20260925 16:31:31.923273 19519 raft_consensus.cc:2781] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:31.923738 19519 raft_consensus.cc:686] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [term 1 LEADER]: Becoming Leader. State: Replica: f84ab829a32b40c2a90a9b8ca1f6d7dc, State: Running, Role: LEADER
I20260925 16:31:31.924194 19519 consensus_queue.cc:237] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } }
I20260925 16:31:31.928670 19045 catalog_manager.cc:5495] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to f84ab829a32b40c2a90a9b8ca1f6d7dc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84ab829a32b40c2a90a9b8ca1f6d7dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 } health_report { overall_health: UNKNOWN } } }
I20260925 16:31:31.946170 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.941830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.0.98.1:43707)
0925 16:31:31.942001 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:31.942007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:31.942025 (+ 18us) server_negotiation.cc:407] Connection header received
0925 16:31:31.942147 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:31.942149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:31.942227 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:31.942346 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:31:31.942639 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.943228 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.944194 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:31.944428 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:31.944526 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:31.944962 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:31:31.945556 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:31:31.945748 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:31.945907 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 16:31:31.945956 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:31:31.954949 19318 raft_consensus.cc:1261] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 1 FOLLOWER]: Refusing update from remote peer f84ab829a32b40c2a90a9b8ca1f6d7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:31.954990 19449 raft_consensus.cc:1261] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 1 FOLLOWER]: Refusing update from remote peer f84ab829a32b40c2a90a9b8ca1f6d7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:31.955922 19519 consensus_queue.cc:1035] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eb3ee0d1fb542b1847358cf7e63eb54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:31.956355 19523 consensus_queue.cc:1035] T 83d0290a1b154a93a3e04f8acc9c65d3 P f84ab829a32b40c2a90a9b8ca1f6d7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3e315f08094d1298d762851de82ff2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:31.966780 19527 mvcc.cc:205] Tried to move back new op lower bound from 7333289541436903424 to 7333289541324734464. Current Snapshot: MvccSnapshot[applied={T|T < 7333289541436903424}]
I20260925 16:31:31.966822 19528 mvcc.cc:205] Tried to move back new op lower bound from 7333289541436903424 to 7333289541324734464. Current Snapshot: MvccSnapshot[applied={T|T < 7333289541436903424}]
I20260925 16:31:31.976660 19044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:31.976891 19044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:31.980072 19044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc0408db4194d56bf0a8fda13ce7e53: Sending DeleteTablet for 3 replicas of tablet 83d0290a1b154a93a3e04f8acc9c65d3
I20260925 16:31:31.981299 19429 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d0290a1b154a93a3e04f8acc9c65d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:31 UTC) from {username='slave'} at 127.0.0.1:48672
I20260925 16:31:31.981299 19298 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d0290a1b154a93a3e04f8acc9c65d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:31 UTC) from {username='slave'} at 127.0.0.1:48952
I20260925 16:31:31.982038 19535 tablet_replica.cc:330] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: stopping tablet replica
I20260925 16:31:31.982048 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d0290a1b154a93a3e04f8acc9c65d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:31 UTC) from {username='slave'} at 127.0.0.1:58492
I20260925 16:31:31.982421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19105
I20260925 16:31:31.982518 19535 raft_consensus.cc:2229] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:31.982980 19535 raft_consensus.cc:2258] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:31.983495 19536 tablet_replica.cc:330] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: stopping tablet replica
I20260925 16:31:31.983953 19536 raft_consensus.cc:2229] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:31.984345 19536 raft_consensus.cc:2258] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:31.986672 19535 ts_tablet_manager.cc:1905] T 83d0290a1b154a93a3e04f8acc9c65d3 P 0eb3ee0d1fb542b1847358cf7e63eb54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:31.986672 19536 ts_tablet_manager.cc:1905] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:31:31.990027 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43707 (error 108)
I20260925 16:31:31.990512 19536 ts_tablet_manager.cc:1918] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:31.990701 19536 log.cc:1196] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/83d0290a1b154a93a3e04f8acc9c65d3
I20260925 16:31:31.991031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19236
I20260925 16:31:31.991076 19536 ts_tablet_manager.cc:1939] T 83d0290a1b154a93a3e04f8acc9c65d3 P 4f3e315f08094d1298d762851de82ff2: Deleting consensus metadata
I20260925 16:31:31.993992 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.993420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43707 (local address 127.0.0.1:58520)
0925 16:31:31.993707 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:31.993870 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 16:31:31.994295 19030 catalog_manager.cc:4584] TS f84ab829a32b40c2a90a9b8ca1f6d7dc (127.0.98.1:43707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83d0290a1b154a93a3e04f8acc9c65d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43707: connect: Connection refused (error 111)
I20260925 16:31:31.994586 19030 catalog_manager.cc:4841] TS 4f3e315f08094d1298d762851de82ff2 (127.0.98.3:44507): tablet 83d0290a1b154a93a3e04f8acc9c65d3 (table test-table [id=caa947afbd6d4109ba5c4bd286d5bafa]) successfully deleted
I20260925 16:31:31.998653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19367
I20260925 16:31:32.000527 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:31.999726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33889 (local address 127.0.0.1:48960)
0925 16:31:32.000347 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:32.000413 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
I20260925 16:31:32.005213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19014
2026-09-25T16:31:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1495 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:32 dist-test-slave-31kt krb5kdc[19541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:32 dist-test-slave-31kt krb5kdc[19541](info): set up 2 sockets
Sep 25 16:31:32 dist-test-slave-31kt krb5kdc[19541](info): commencing operation
krb5kdc: starting...
W20260925 16:31:32.998698 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:33 dist-test-slave-31kt krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:33.138084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46349
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:33.271095 19554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:33.271514 19554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:33.271581 19554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:33.276371 19554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:33.276489 19554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:33.276520 19554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:33.276540 19554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:33.276561 19554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:33.283246 19554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:33.284889 19554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:33.286311 19554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:33.297626 19560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:33.297590 19562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:33.297626 19559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:33.297842 19554 server_base.cc:1034] running on GCE node
I20260925 16:31:33.298816 19554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:33.300413 19554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:33.301631 19554 hybrid_clock.cc:648] HybridClock initialized: now 1790353893301577 us; error 83 us; skew 500 ppm
Sep 25 16:31:33 dist-test-slave-31kt krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:33.305485 19554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:33.306913 19554 webserver.cc:460] Webserver started at http://127.0.98.62:41011/ using document root <none> and password file <none>
I20260925 16:31:33.307787 19554 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:33.307878 19554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:33.308136 19554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:33.311103 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "2958b542a11d4e048bc63caf5688d4f3"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "6df1748033809ae57aad2404bda1569c"
server_key_iv: "cad30eb311674422735b7204aced6293"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.311885 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "2958b542a11d4e048bc63caf5688d4f3"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "6df1748033809ae57aad2404bda1569c"
server_key_iv: "cad30eb311674422735b7204aced6293"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.317417 19554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:31:33.321254 19569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:33.323136 19554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:33.323344 19554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "2958b542a11d4e048bc63caf5688d4f3"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "6df1748033809ae57aad2404bda1569c"
server_key_iv: "cad30eb311674422735b7204aced6293"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.323499 19554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:33.340524 19554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:33.345086 19554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:33.345340 19554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:33.353822 19554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46349
I20260925 16:31:33.353858 19621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46349 every 8 connection(s)
I20260925 16:31:33.355211 19554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:33.357437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19554
I20260925 16:31:33.357614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:33.358007 392 external_mini_cluster.cc:1428] Setting key 47db5eaa19aab0cf50870e2e978b7cb6
I20260925 16:31:33.359810 19622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:33.368328 19622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3: Bootstrap starting.
Sep 25 16:31:33 dist-test-slave-31kt krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:33.371884 19622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:33.372907 19622 log.cc:824] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:33.375226 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:33.360074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.98.62:46349)
0925 16:31:33.360866 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:33.360886 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:33.360952 (+ 66us) server_negotiation.cc:407] Connection header received
0925 16:31:33.361739 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:33.361807 (+ 68us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:33.362265 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:33.362657 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:33.363574 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:33.364684 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:33.365628 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:33.366255 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:33.366442 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:33.369453 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:33.369460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:33.369480 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:33.369524 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:33.371678 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:33.372269 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:33.372276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:33.372282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:33.372429 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:33.372725 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:33.372729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:33.372730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:33.373375 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:33.373590 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:33.373975 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 16:31:33.374329 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":234,"threads_started":1}
I20260925 16:31:33.375862 19622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3: No bootstrap required, opened a new log
I20260925 16:31:33.380430 19622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958b542a11d4e048bc63caf5688d4f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46349 } }
I20260925 16:31:33.380836 19622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:33.380904 19622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2958b542a11d4e048bc63caf5688d4f3, State: Initialized, Role: FOLLOWER
I20260925 16:31:33.381484 19622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958b542a11d4e048bc63caf5688d4f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46349 } }
I20260925 16:31:33.381650 19622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:33.381724 19622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:33.381861 19622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:33.383204 19622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958b542a11d4e048bc63caf5688d4f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46349 } }
I20260925 16:31:33.383677 19622 leader_election.cc:302] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2958b542a11d4e048bc63caf5688d4f3; no voters:
I20260925 16:31:33.384182 19622 leader_election.cc:288] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:33.384426 19627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:33.384907 19627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [term 1 LEADER]: Becoming Leader. State: Replica: 2958b542a11d4e048bc63caf5688d4f3, State: Running, Role: LEADER
I20260925 16:31:33.385344 19627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958b542a11d4e048bc63caf5688d4f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46349 } }
I20260925 16:31:33.386010 19622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:33.388144 19629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2958b542a11d4e048bc63caf5688d4f3. Latest consensus state: current_term: 1 leader_uuid: "2958b542a11d4e048bc63caf5688d4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958b542a11d4e048bc63caf5688d4f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46349 } } }
I20260925 16:31:33.388116 19628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2958b542a11d4e048bc63caf5688d4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958b542a11d4e048bc63caf5688d4f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46349 } } }
I20260925 16:31:33.388405 19629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:33.388435 19628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:33.389279 19636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:33.394408 19636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:33.403331 19636 catalog_manager.cc:1347] Generated new cluster ID: 5e340e8f3f144d7c9b8ec243b57ffef3
I20260925 16:31:33.403447 19636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:33.417119 19636 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:33.418505 19636 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:33.429787 19636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3: Generated new TSK 0
I20260925 16:31:33.430913 19636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 16:31:33.465369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46349
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:33.605695 19648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:33.606177 19648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:33.606251 19648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:33.611116 19648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:33.611229 19648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:33.611330 19648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:33.617923 19648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:33.619524 19648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:33.620893 19648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:33.633327 19656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:33.633339 19653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:33.633512 19654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:33.633884 19648 server_base.cc:1034] running on GCE node
I20260925 16:31:33.634466 19648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:33.635371 19648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:33.636582 19648 hybrid_clock.cc:648] HybridClock initialized: now 1790353893636567 us; error 64 us; skew 500 ppm
Sep 25 16:31:33 dist-test-slave-31kt krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:33.640331 19648 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 16:31:33.641733 19648 webserver.cc:460] Webserver started at http://127.0.98.1:37443/ using document root <none> and password file <none>
I20260925 16:31:33.642596 19648 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:33.642688 19648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:33.642957 19648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:33.645823 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "6408aeb3c202490f85c0410d73c2a4e2"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "b08d18862e614c3fff9c4953ed7909ea"
server_key_iv: "d641c4c34d23b05ef6f61b6a6e9323e7"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.646545 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6408aeb3c202490f85c0410d73c2a4e2"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "b08d18862e614c3fff9c4953ed7909ea"
server_key_iv: "d641c4c34d23b05ef6f61b6a6e9323e7"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.651902 19648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:31:33.656144 19663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:33.658350 19648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:31:33.658569 19648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "6408aeb3c202490f85c0410d73c2a4e2"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "b08d18862e614c3fff9c4953ed7909ea"
server_key_iv: "d641c4c34d23b05ef6f61b6a6e9323e7"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.658751 19648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:33.706789 19648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:33.711308 19648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:33.711561 19648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:33.712363 19648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:33.713721 19648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:33.713848 19648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 16:31:33.713930 19648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:33.713956 19648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:33.725798 19648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39889
I20260925 16:31:33.725835 19776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39889 every 8 connection(s)
I20260925 16:31:33.727032 19648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:33.735600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19648
I20260925 16:31:33.735769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:31:33.736107 392 external_mini_cluster.cc:1428] Setting key 9aa732ac044b6615d5b66379c75323c0
Sep 25 16:31:33 dist-test-slave-31kt krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:33.745587 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:33.729691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57459 (local address 127.0.98.62:46349)
0925 16:31:33.729938 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:33.729947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:33.731274 (+ 1327us) server_negotiation.cc:407] Connection header received
0925 16:31:33.732792 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:33.732795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:33.732861 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:33.732937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:33.734413 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:33.734970 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:33.736573 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:33.736833 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:33.736953 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:33.740348 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:33.740354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:33.740357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:33.740392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:33.742155 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:33.742895 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:33.742899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:33.742902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:33.743015 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:33.743494 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:33.743498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:33.743500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:33.743806 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:33.743943 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:33.745039 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0925 16:31:33.745245 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 16:31:33.746755 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:33.730272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46349 (local address 127.0.98.1:57459)
0925 16:31:33.731056 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:33.731123 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:33.732531 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:33.733167 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:33.733178 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:33.733654 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:33.734225 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:33.734240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:33.735103 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:33.735106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:33.736451 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:33.736465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:33.737115 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:33.737123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:33.737492 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:33.738365 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:33.738387 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:33.740070 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:33.742347 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:33.742354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:33.742371 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:33.742745 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:33.743160 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:33.743163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:33.743166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:33.743368 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:33.744010 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:33.744020 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:33.744568 (+ 548us) client_negotiation.cc:769] Sending connection context
0925 16:31:33.744863 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 16:31:33.745656 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":215,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 16:31:33.748725 19777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46349
I20260925 16:31:33.749217 19777 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 16:31:33.750176 19777 heartbeater.cc:511] Master 127.0.98.62:46349 requested a full tablet report, sending...
I20260925 16:31:33.753077 19585 ts_manager.cc:194] Registered new tserver with Master: 6408aeb3c202490f85c0410d73c2a4e2 (127.0.98.1:39889)
I20260925 16:31:33.754930 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 16:31:33.769932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46349
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:33.905689 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:33.906142 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:33.906198 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:33.911055 19782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:33.911150 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:33.911223 19782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:33.917879 19782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:33.919440 19782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:33.920820 19782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:33.933674 19788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:33.933691 19787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:33.933909 19782 server_base.cc:1034] running on GCE node
W20260925 16:31:33.934156 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:33.934738 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:33.935688 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:33.936906 19782 hybrid_clock.cc:648] HybridClock initialized: now 1790353893936874 us; error 59 us; skew 500 ppm
Sep 25 16:31:33 dist-test-slave-31kt krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:33.940804 19782 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 16:31:33.942273 19782 webserver.cc:460] Webserver started at http://127.0.98.2:35747/ using document root <none> and password file <none>
I20260925 16:31:33.943034 19782 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:33.943121 19782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:33.943370 19782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:33.946208 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "6642e61d9a284c3cbc9bbad31affa125"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "2ce4e35dcc7c82e3a5a2eb33308d6c3b"
server_key_iv: "6be998d770d4256de1f170a3f6669290"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.946902 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6642e61d9a284c3cbc9bbad31affa125"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "2ce4e35dcc7c82e3a5a2eb33308d6c3b"
server_key_iv: "6be998d770d4256de1f170a3f6669290"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.951838 19782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:33.955327 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:33.957239 19782 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:33.957437 19782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "6642e61d9a284c3cbc9bbad31affa125"
format_stamp: "Formatted at 2026-09-25 16:31:33 on dist-test-slave-31kt"
server_key: "2ce4e35dcc7c82e3a5a2eb33308d6c3b"
server_key_iv: "6be998d770d4256de1f170a3f6669290"
server_key_version: "encryptionkey@0"
I20260925 16:31:33.957595 19782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:33.982914 19782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:33.987534 19782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:33.987783 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:33.988581 19782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:33.989743 19782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:33.989853 19782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:33.989913 19782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:33.989930 19782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:34.002321 19782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33771
I20260925 16:31:34.002444 19910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33771 every 8 connection(s)
I20260925 16:31:34.003580 19782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:34.009351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19782
I20260925 16:31:34.009519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:31:34.009892 392 external_mini_cluster.cc:1428] Setting key 06cec977e656a8c98f88c1191aa74611
Sep 25 16:31:34 dist-test-slave-31kt krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 16:31:34.023125 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.006264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58427 (local address 127.0.98.62:46349)
0925 16:31:34.006433 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.006438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.007632 (+ 1194us) server_negotiation.cc:407] Connection header received
0925 16:31:34.009292 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.009297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.009368 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.009471 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:34.011396 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.012092 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.014203 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.014592 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.014797 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.018389 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:34.018394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:34.018398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:34.018450 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:34.020334 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:34.021100 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:34.021104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:34.021106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:34.021219 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:34.021725 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:34.021728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:34.021730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:34.021986 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:34.022075 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.022844 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.022978 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 16:31:34.024286 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.006609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46349 (local address 127.0.98.2:58427)
0925 16:31:34.007350 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.007427 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.008924 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.009751 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.009796 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.010416 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:34.011184 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.011204 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.012272 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.012276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.013935 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.013964 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.014887 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.014891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.015149 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.015907 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:34.015974 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:34.018083 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:34.020518 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:34.020524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:34.020548 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:34.020938 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:34.021350 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:34.021353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:34.021355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:34.021603 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:34.022104 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:34.022110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:34.022520 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.022730 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.023122 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":181,"threads_started":1}
I20260925 16:31:34.025936 19911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46349
I20260925 16:31:34.026239 19911 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:34.026873 19911 heartbeater.cc:511] Master 127.0.98.62:46349 requested a full tablet report, sending...
I20260925 16:31:34.028265 19585 ts_manager.cc:194] Registered new tserver with Master: 6642e61d9a284c3cbc9bbad31affa125 (127.0.98.2:33771)
I20260925 16:31:34.028999 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 16:31:34.041937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46349
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:34.176190 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:34.176512 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:34.176573 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:34.180928 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:34.181012 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:34.181100 19916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:34.188707 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:34.190537 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:34.191773 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:34.203691 19922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:34.203840 19916 server_base.cc:1034] running on GCE node
W20260925 16:31:34.203718 19921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:34.203720 19924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:34.204596 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:34.205404 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:34.206643 19916 hybrid_clock.cc:648] HybridClock initialized: now 1790353894206613 us; error 52 us; skew 500 ppm
Sep 25 16:31:34 dist-test-slave-31kt krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:34.210901 19916 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 16:31:34.212364 19916 webserver.cc:460] Webserver started at http://127.0.98.3:33013/ using document root <none> and password file <none>
I20260925 16:31:34.213474 19916 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:34.213608 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:34.213917 19916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:34.217080 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "af802f6ed5514a75b3d9841c9d91a293"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "bfeb2f8f3e61f42819b3dc5080786cea"
server_key_iv: "2e71eb6a5a3eb2068034a5e4eaf9552f"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.218025 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af802f6ed5514a75b3d9841c9d91a293"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "bfeb2f8f3e61f42819b3dc5080786cea"
server_key_iv: "2e71eb6a5a3eb2068034a5e4eaf9552f"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.225549 19916 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260925 16:31:34.229403 19931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:34.231320 19916 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:34.231539 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "af802f6ed5514a75b3d9841c9d91a293"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "bfeb2f8f3e61f42819b3dc5080786cea"
server_key_iv: "2e71eb6a5a3eb2068034a5e4eaf9552f"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.231688 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:34.255369 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:34.259682 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:34.259910 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:34.260599 19916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:34.261648 19916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:34.261703 19916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:34.261878 19916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:34.261910 19916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:34.272113 20044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39153 every 8 connection(s)
I20260925 16:31:34.272168 19916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39153
I20260925 16:31:34.273708 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:34.281832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 19916
I20260925 16:31:34.282008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:31:34.282368 392 external_mini_cluster.cc:1428] Setting key 95c105a5144bde023399f67aaa5246c0
Sep 25 16:31:34 dist-test-slave-31kt krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:34.292200 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.276581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60167 (local address 127.0.98.62:46349)
0925 16:31:34.276841 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.276846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.278031 (+ 1185us) server_negotiation.cc:407] Connection header received
0925 16:31:34.279608 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.279612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.279683 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.279762 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:34.281197 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.281814 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.283214 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.283448 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.283543 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.287149 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:34.287153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:34.287155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:34.287194 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:34.289286 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:34.290451 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:34.290455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:34.290456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:34.290584 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:34.290985 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:34.290988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:34.290989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:34.291199 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:34.291296 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.291865 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.292027 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260925 16:31:34.293078 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.276990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46349 (local address 127.0.98.3:60167)
0925 16:31:34.277743 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.277886 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.279312 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.279992 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.280001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.280428 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:34.281005 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.281021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.281951 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.281955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.283083 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.283095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.283612 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.283616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.283844 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.284598 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:31:34.284616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:34.286837 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:34.289537 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:34.289548 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:34.289572 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:34.290266 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:34.290693 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:31:34.290696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:31:34.290699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:31:34.290876 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:31:34.291307 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:34.291315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:34.291610 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.291796 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.292169 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":181,"threads_started":1}
I20260925 16:31:34.294706 20045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46349
I20260925 16:31:34.295140 20045 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:34.295897 20045 heartbeater.cc:511] Master 127.0.98.62:46349 requested a full tablet report, sending...
I20260925 16:31:34.297493 19585 ts_manager.cc:194] Registered new tserver with Master: af802f6ed5514a75b3d9841c9d91a293 (127.0.98.3:39153)
I20260925 16:31:34.298166 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:31:34.298523 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60167
I20260925 16:31:34.310616 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.301046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.98.62:46349)
0925 16:31:34.301238 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.301244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.301533 (+ 289us) server_negotiation.cc:407] Connection header received
0925 16:31:34.301715 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.301718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.301847 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.301939 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:34.302811 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.303543 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.304580 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.304866 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.304970 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.306645 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:34.306649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:34.306657 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:34.306692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:34.308433 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:34.308995 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:34.309000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:34.309002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:34.309106 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:34.309495 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:34.309497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:34.309498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:34.309799 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:34.309929 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.310301 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.310443 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 16:31:34.314076 19585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:31:34.317355 19585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:31:34.338105 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.330867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.0.98.1:39889)
0925 16:31:34.331297 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.331302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.331325 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:31:34.331411 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.331417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.331649 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.331798 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:34.332278 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.333271 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.335507 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.336557 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.336669 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.336896 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.337415 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.337561 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":102,"threads_started":1}
I20260925 16:31:34.338860 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.329333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39889 (local address 127.0.0.1:44224)
0925 16:31:34.329914 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.329993 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.330179 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.331897 (+ 1718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.331906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.331946 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:34.332134 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.332142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.333399 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.333403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.334915 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.334928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.336972 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.336986 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.337160 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.337206 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.338525 (+ 1319us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.338718 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":27,"thread_start_us":132,"threads_started":1}
I20260925 16:31:34.339900 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.330387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33771 (local address 127.0.0.1:40682)
0925 16:31:34.333495 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.333511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.333611 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.334507 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.334510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.334532 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:34.334990 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.334996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.336348 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.336352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.337479 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.337490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.339401 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.339405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.339497 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.339526 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.339608 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.339706 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3032,"mutex_wait_us":2,"thread_start_us":147,"threads_started":1}
I20260925 16:31:34.340034 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.330709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.0.98.2:33771)
0925 16:31:34.331194 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.331209 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.333715 (+ 2506us) server_negotiation.cc:407] Connection header received
0925 16:31:34.333850 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.333857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.334125 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.334304 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:34.335126 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.336192 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.337676 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.339020 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.339131 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.339325 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.339697 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.339854 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":202,"threads_started":1}
I20260925 16:31:34.341176 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.333528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39153 (local address 127.0.0.1:43490)
0925 16:31:34.334401 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.334416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.334524 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.335505 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.335508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.335528 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:34.335661 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.335668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.337010 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.337014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.338411 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.338423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.339740 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.339744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.339839 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.339859 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.339923 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.339994 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260925 16:31:34.341935 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.334147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.0.98.3:39153)
0925 16:31:34.334721 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.334734 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.334757 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:31:34.334845 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.334851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.335114 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.335298 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:34.335794 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.336870 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.338603 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.339441 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.339546 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.339681 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.341550 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.341721 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":174,"threads_started":1}
I20260925 16:31:34.343165 19711 tablet_service.cc:1467] Processing CreateTablet for tablet 3695f54a268244c8ac2a78732ac612ae (DEFAULT_TABLE table=test-table [id=4a2d149375d248e28438392491b664ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:34.344090 19845 tablet_service.cc:1467] Processing CreateTablet for tablet 3695f54a268244c8ac2a78732ac612ae (DEFAULT_TABLE table=test-table [id=4a2d149375d248e28438392491b664ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:34.344857 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f54a268244c8ac2a78732ac612ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:34.345368 19845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f54a268244c8ac2a78732ac612ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:34.345646 19979 tablet_service.cc:1467] Processing CreateTablet for tablet 3695f54a268244c8ac2a78732ac612ae (DEFAULT_TABLE table=test-table [id=4a2d149375d248e28438392491b664ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:34.347023 19979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3695f54a268244c8ac2a78732ac612ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:34.352648 20062 tablet_bootstrap.cc:492] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Bootstrap starting.
I20260925 16:31:34.355228 20062 tablet_bootstrap.cc:654] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:34.355223 20063 tablet_bootstrap.cc:492] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2: Bootstrap starting.
I20260925 16:31:34.356448 20062 log.cc:824] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:34.358302 20063 tablet_bootstrap.cc:654] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:34.358855 20062 tablet_bootstrap.cc:492] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: No bootstrap required, opened a new log
I20260925 16:31:34.359231 20062 ts_tablet_manager.cc:1397] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 16:31:34.359526 20063 log.cc:824] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:34.362376 20065 tablet_bootstrap.cc:492] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Bootstrap starting.
I20260925 16:31:34.362507 20063 tablet_bootstrap.cc:492] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2: No bootstrap required, opened a new log
I20260925 16:31:34.362958 20063 ts_tablet_manager.cc:1397] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 16:31:34.363252 20062 raft_consensus.cc:348] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.363703 20062 raft_consensus.cc:374] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:34.363772 20062 raft_consensus.cc:729] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642e61d9a284c3cbc9bbad31affa125, State: Initialized, Role: FOLLOWER
I20260925 16:31:34.364454 20062 consensus_queue.cc:260] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.365342 20065 tablet_bootstrap.cc:654] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:34.365661 20062 ts_tablet_manager.cc:1428] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 16:31:34.366196 19911 heartbeater.cc:503] Master 127.0.98.62:46349 was elected leader, sending a full tablet report...
I20260925 16:31:34.368126 20065 log.cc:824] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:34.369346 20063 raft_consensus.cc:348] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.369912 20063 raft_consensus.cc:374] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:34.370011 20063 raft_consensus.cc:729] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6408aeb3c202490f85c0410d73c2a4e2, State: Initialized, Role: FOLLOWER
I20260925 16:31:34.370743 20063 consensus_queue.cc:260] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.371344 20065 tablet_bootstrap.cc:492] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: No bootstrap required, opened a new log
I20260925 16:31:34.371694 20065 ts_tablet_manager.cc:1397] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 16:31:34.372330 20063 ts_tablet_manager.cc:1428] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 16:31:34.372399 19777 heartbeater.cc:503] Master 127.0.98.62:46349 was elected leader, sending a full tablet report...
I20260925 16:31:34.375859 20065 raft_consensus.cc:348] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.376250 20065 raft_consensus.cc:374] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:34.376333 20065 raft_consensus.cc:729] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af802f6ed5514a75b3d9841c9d91a293, State: Initialized, Role: FOLLOWER
I20260925 16:31:34.376832 20065 consensus_queue.cc:260] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.377941 20045 heartbeater.cc:503] Master 127.0.98.62:46349 was elected leader, sending a full tablet report...
I20260925 16:31:34.378206 20065 ts_tablet_manager.cc:1428] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 16:31:34.378648 20070 raft_consensus.cc:482] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:34.378841 20070 raft_consensus.cc:504] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.380089 20070 leader_election.cc:288] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6408aeb3c202490f85c0410d73c2a4e2 (127.0.98.1:39889), 6642e61d9a284c3cbc9bbad31affa125 (127.0.98.2:33771)
I20260925 16:31:34.385049 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.380593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58839 (local address 127.0.98.1:39889)
0925 16:31:34.380754 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.380759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.380773 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:31:34.380834 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.380837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.380889 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.380970 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:34.381460 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.382242 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.383447 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.384349 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.384475 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.384553 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.384830 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.384912 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 16:31:34.385087 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.380314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39889 (local address 127.0.98.3:58839)
0925 16:31:34.380485 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.380507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.380620 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.381122 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.381125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.381183 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:34.381322 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.381331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.382353 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.382356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.383327 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.383339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.384509 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.384513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.384644 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.384677 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.384780 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.384888 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"spinlock_wait_cycles":14720}
I20260925 16:31:34.385926 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.380614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33771 (local address 127.0.98.3:50643)
0925 16:31:34.381011 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.381027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:34.381200 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:34.381458 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:34.381460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:34.381478 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:34.381616 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.381622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.382724 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.382728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.384229 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:34.384248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.385391 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.385395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:34.385512 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.385533 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 16:31:34.385599 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 16:31:34.385727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"spinlock_wait_cycles":19072,"thread_start_us":162,"threads_started":1}
I20260925 16:31:34.385987 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.380783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50643 (local address 127.0.98.2:33771)
0925 16:31:34.380952 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.380957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.381124 (+ 167us) server_negotiation.cc:407] Connection header received
0925 16:31:34.381316 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.381319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.381371 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.381434 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:34.381720 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.382514 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.384481 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.385133 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.385248 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.385326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.385658 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.385795 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:31:34.386046 19731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f54a268244c8ac2a78732ac612ae" candidate_uuid: "af802f6ed5514a75b3d9841c9d91a293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6408aeb3c202490f85c0410d73c2a4e2" is_pre_election: true
I20260925 16:31:34.386449 19731 raft_consensus.cc:2454] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af802f6ed5514a75b3d9841c9d91a293 in term 0.
I20260925 16:31:34.386628 19865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f54a268244c8ac2a78732ac612ae" candidate_uuid: "af802f6ed5514a75b3d9841c9d91a293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642e61d9a284c3cbc9bbad31affa125" is_pre_election: true
I20260925 16:31:34.387027 19865 raft_consensus.cc:2454] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af802f6ed5514a75b3d9841c9d91a293 in term 0.
I20260925 16:31:34.387073 19932 leader_election.cc:302] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6408aeb3c202490f85c0410d73c2a4e2, af802f6ed5514a75b3d9841c9d91a293; no voters:
I20260925 16:31:34.387516 20071 raft_consensus.cc:2781] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:31:34.387696 20071 raft_consensus.cc:482] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:31:34.387769 20071 raft_consensus.cc:3037] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:34.389015 20071 raft_consensus.cc:504] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.389443 20071 leader_election.cc:288] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [CANDIDATE]: Term 1 election: Requested vote from peers 6408aeb3c202490f85c0410d73c2a4e2 (127.0.98.1:39889), 6642e61d9a284c3cbc9bbad31affa125 (127.0.98.2:33771)
I20260925 16:31:34.390096 19865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f54a268244c8ac2a78732ac612ae" candidate_uuid: "af802f6ed5514a75b3d9841c9d91a293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642e61d9a284c3cbc9bbad31affa125"
I20260925 16:31:34.390084 19731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3695f54a268244c8ac2a78732ac612ae" candidate_uuid: "af802f6ed5514a75b3d9841c9d91a293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6408aeb3c202490f85c0410d73c2a4e2"
I20260925 16:31:34.390277 19865 raft_consensus.cc:3037] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:34.390277 19731 raft_consensus.cc:3037] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:34.392035 19731 raft_consensus.cc:2454] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af802f6ed5514a75b3d9841c9d91a293 in term 1.
I20260925 16:31:34.392038 19865 raft_consensus.cc:2454] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af802f6ed5514a75b3d9841c9d91a293 in term 1.
I20260925 16:31:34.392478 19932 leader_election.cc:302] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6642e61d9a284c3cbc9bbad31affa125, af802f6ed5514a75b3d9841c9d91a293; no voters:
I20260925 16:31:34.392714 20071 raft_consensus.cc:2781] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:34.392830 20071 raft_consensus.cc:686] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 1 LEADER]: Becoming Leader. State: Replica: af802f6ed5514a75b3d9841c9d91a293, State: Running, Role: LEADER
I20260925 16:31:34.393260 20071 consensus_queue.cc:237] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } }
I20260925 16:31:34.397503 19586 catalog_manager.cc:5495] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 reported cstate change: term changed from 0 to 1, leader changed from <none> to af802f6ed5514a75b3d9841c9d91a293 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af802f6ed5514a75b3d9841c9d91a293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af802f6ed5514a75b3d9841c9d91a293" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39153 } health_report { overall_health: HEALTHY } } }
I20260925 16:31:34.418213 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.414551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.0.98.3:39153)
0925 16:31:34.414716 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.414721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.414738 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:31:34.414894 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.414897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.414961 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.415061 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:31:34.415305 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.415822 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.416672 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.416887 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.416973 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.417283 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:31:34.417792 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:31:34.417914 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.417974 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.418043 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:31:34.426195 19865 raft_consensus.cc:1261] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 1 FOLLOWER]: Refusing update from remote peer af802f6ed5514a75b3d9841c9d91a293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:34.426195 19731 raft_consensus.cc:1261] T 3695f54a268244c8ac2a78732ac612ae P 6408aeb3c202490f85c0410d73c2a4e2 [term 1 FOLLOWER]: Refusing update from remote peer af802f6ed5514a75b3d9841c9d91a293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:34.427388 20070 consensus_queue.cc:1035] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6408aeb3c202490f85c0410d73c2a4e2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:34.427704 20071 consensus_queue.cc:1035] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6642e61d9a284c3cbc9bbad31affa125" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:34.437227 20075 mvcc.cc:205] Tried to move back new op lower bound from 7333289551558561792 to 7333289551438274560. Current Snapshot: MvccSnapshot[applied={T|T < 7333289551558561792}]
I20260925 16:31:34.438606 20077 mvcc.cc:205] Tried to move back new op lower bound from 7333289551558561792 to 7333289551438274560. Current Snapshot: MvccSnapshot[applied={T|T < 7333289551558561792}]
I20260925 16:31:34.440830 20076 mvcc.cc:205] Tried to move back new op lower bound from 7333289551558561792 to 7333289551438274560. Current Snapshot: MvccSnapshot[applied={T|T < 7333289551558561792}]
I20260925 16:31:34.449929 19586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:34.450130 19586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:34.453473 19586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2958b542a11d4e048bc63caf5688d4f3: Sending DeleteTablet for 3 replicas of tablet 3695f54a268244c8ac2a78732ac612ae
I20260925 16:31:34.454873 19979 tablet_service.cc:1514] Processing DeleteTablet for tablet 3695f54a268244c8ac2a78732ac612ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260925 16:31:34.455020 19711 tablet_service.cc:1514] Processing DeleteTablet for tablet 3695f54a268244c8ac2a78732ac612ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44224
I20260925 16:31:34.455283 19845 tablet_service.cc:1514] Processing DeleteTablet for tablet 3695f54a268244c8ac2a78732ac612ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40682
I20260925 16:31:34.455690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19648
I20260925 16:31:34.456746 20085 tablet_replica.cc:330] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: stopping tablet replica
I20260925 16:31:34.456961 20087 tablet_replica.cc:330] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: stopping tablet replica
I20260925 16:31:34.457218 20085 raft_consensus.cc:2229] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:31:34.457336 20087 raft_consensus.cc:2229] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:34.457711 20087 raft_consensus.cc:2258] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:34.457791 20085 raft_consensus.cc:2258] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:34.459863 20085 ts_tablet_manager.cc:1905] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:34.460392 20087 ts_tablet_manager.cc:1905] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:34.463521 20085 ts_tablet_manager.cc:1918] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:34.463658 20085 log.cc:1196] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/3695f54a268244c8ac2a78732ac612ae
I20260925 16:31:34.463843 20087 ts_tablet_manager.cc:1918] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:34.464007 20085 ts_tablet_manager.cc:1939] T 3695f54a268244c8ac2a78732ac612ae P af802f6ed5514a75b3d9841c9d91a293: Deleting consensus metadata
I20260925 16:31:34.464140 20087 log.cc:1196] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/3695f54a268244c8ac2a78732ac612ae
I20260925 16:31:34.464484 20087 ts_tablet_manager.cc:1939] T 3695f54a268244c8ac2a78732ac612ae P 6642e61d9a284c3cbc9bbad31affa125: Deleting consensus metadata
I20260925 16:31:34.465135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19782
W20260925 16:31:34.465246 19570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39889 (error 108)
I20260925 16:31:34.467835 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.467393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39889 (local address 127.0.0.1:44226)
0925 16:31:34.467587 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.467716 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 16:31:34.467844 19573 catalog_manager.cc:4841] TS af802f6ed5514a75b3d9841c9d91a293 (127.0.98.3:39153): tablet 3695f54a268244c8ac2a78732ac612ae (table test-table [id=4a2d149375d248e28438392491b664ef]) successfully deleted
W20260925 16:31:34.468130 19570 catalog_manager.cc:4584] TS 6408aeb3c202490f85c0410d73c2a4e2 (127.0.98.1:39889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3695f54a268244c8ac2a78732ac612ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39889: connect: Connection refused (error 111)
I20260925 16:31:34.473444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19916
I20260925 16:31:34.473930 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.473408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33771 (local address 127.0.0.1:40692)
0925 16:31:34.473552 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:34.473614 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 16:31:34.480515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 19554
2026-09-25T16:31:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2475 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T16:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:34Z Disabled control of system clock
I20260925 16:31:34.515789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:31:34.654637 20092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:34.655026 20092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:34.655067 20092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:34.659858 20092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:34.659966 20092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:34.659981 20092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:34.659999 20092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:34.660017 20092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:34.666141 20092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:34.667703 20092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:34.669559 20092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:34.680312 20097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:34.680266 20098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:34.680266 20100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:34.680806 20092 server_base.cc:1034] running on GCE node
I20260925 16:31:34.681706 20092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:34.683269 20092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:34.684527 20092 hybrid_clock.cc:648] HybridClock initialized: now 1790353894684464 us; error 92 us; skew 500 ppm
I20260925 16:31:34.687029 20092 webserver.cc:460] Webserver started at http://127.0.98.62:34435/ using document root <none> and password file <none>
I20260925 16:31:34.687723 20092 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:34.687814 20092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:34.688051 20092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:34.690383 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "2fc6a44bedd44a45a00fddf2c37e0994"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "b31fea230bec089423d60b750405c19f"
server_key_iv: "4ec99bcb365ef31f60d3f1d1c5cc6da0"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.690997 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fc6a44bedd44a45a00fddf2c37e0994"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "b31fea230bec089423d60b750405c19f"
server_key_iv: "4ec99bcb365ef31f60d3f1d1c5cc6da0"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.695675 20092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:34.699044 20106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:34.700784 20092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:31:34.700975 20092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "2fc6a44bedd44a45a00fddf2c37e0994"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "b31fea230bec089423d60b750405c19f"
server_key_iv: "4ec99bcb365ef31f60d3f1d1c5cc6da0"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.701112 20092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:34.716920 20092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:34.717687 20092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:34.717911 20092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:34.725533 20092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40621
I20260925 16:31:34.725587 20158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40621 every 8 connection(s)
I20260925 16:31:34.726693 20092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:34.731077 20159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:34.735059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20092
I20260925 16:31:34.735191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:34.735493 392 external_mini_cluster.cc:1428] Setting key 9935c00921c622be09fc215f2e2febb5
I20260925 16:31:34.739612 20159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994: Bootstrap starting.
I20260925 16:31:34.743335 20159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:34.744445 20159 log.cc:824] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:34.747776 20159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994: No bootstrap required, opened a new log
I20260925 16:31:34.748960 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:34.737411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.0.98.62:40621)
0925 16:31:34.738438 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:34.738468 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:34.738540 (+ 72us) server_negotiation.cc:407] Connection header received
0925 16:31:34.739610 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:34.739637 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:34.740274 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:34.740783 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:34.741438 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.743072 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.744313 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:34.745118 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:34.745305 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:34.746407 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:34.746415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:34.746435 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:34.747025 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:34.747118 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:34.747147 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:34.747270 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:34.747490 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 16:31:34.747826 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":171,"threads_started":1}
I20260925 16:31:34.752714 20159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260925 16:31:34.753036 20159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:34.753073 20159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc6a44bedd44a45a00fddf2c37e0994, State: Initialized, Role: FOLLOWER
I20260925 16:31:34.753571 20159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260925 16:31:34.753734 20159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:34.753834 20159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:34.753983 20159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:34.755192 20159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260925 16:31:34.755632 20159 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fc6a44bedd44a45a00fddf2c37e0994; no voters:
I20260925 16:31:34.756032 20159 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:34.756222 20164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:34.756538 20164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc6a44bedd44a45a00fddf2c37e0994, State: Running, Role: LEADER
I20260925 16:31:34.756973 20164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260925 16:31:34.757653 20159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:34.758867 20166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc6a44bedd44a45a00fddf2c37e0994. Latest consensus state: current_term: 1 leader_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260925 16:31:34.759052 20166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:34.759384 20165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc6a44bedd44a45a00fddf2c37e0994" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260925 16:31:34.759486 20165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:34.759544 20173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:34.763001 20173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:34.771399 20173 catalog_manager.cc:1347] Generated new cluster ID: 48abe81ba19c46c999664f1029291c9f
I20260925 16:31:34.771508 20173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:34.788470 20173 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:34.789521 20173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:34.795101 20173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994: Generated new TSK 0
I20260925 16:31:34.796048 20173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:34.810667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 16:31:34.948504 20183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:34.948836 20183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:34.948890 20183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:34.953410 20183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:34.953503 20183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:34.953563 20183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 16:31:34.959821 20183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:34.961628 20183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:34.963449 20183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:34.975028 20191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:34.975028 20189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:34.975162 20183 server_base.cc:1034] running on GCE node
W20260925 16:31:34.975383 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:34.975901 20183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:34.976670 20183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:34.977873 20183 hybrid_clock.cc:648] HybridClock initialized: now 1790353894977848 us; error 44 us; skew 500 ppm
I20260925 16:31:34.980156 20183 webserver.cc:460] Webserver started at http://127.0.98.1:32901/ using document root <none> and password file <none>
I20260925 16:31:34.980801 20183 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:34.980876 20183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:34.981078 20183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:34.983400 20183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data/instance:
uuid: "01d62f29ae4e44499f76bd679ab84f66"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "5eaacdab141afc9bc60d659c60b9f390"
server_key_iv: "93825ce2846207f17d58f5d59c270377"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.984028 20183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01d62f29ae4e44499f76bd679ab84f66"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "5eaacdab141afc9bc60d659c60b9f390"
server_key_iv: "93825ce2846207f17d58f5d59c270377"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.988607 20183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:31:34.992045 20197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:34.994377 20183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:34.994578 20183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal
uuid: "01d62f29ae4e44499f76bd679ab84f66"
format_stamp: "Formatted at 2026-09-25 16:31:34 on dist-test-slave-31kt"
server_key: "5eaacdab141afc9bc60d659c60b9f390"
server_key_iv: "93825ce2846207f17d58f5d59c270377"
server_key_version: "encryptionkey@0"
I20260925 16:31:34.994724 20183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:35.012281 20183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:35.013315 20183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:35.013543 20183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:35.014377 20183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:35.015658 20183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:35.015715 20183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.015784 20183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:35.015808 20183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.027659 20183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34519
I20260925 16:31:35.027724 20310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34519 every 8 connection(s)
I20260925 16:31:35.028759 20183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/data/info.pb
I20260925 16:31:35.029641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20183
I20260925 16:31:35.029801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-0/wal/instance
I20260925 16:31:35.030112 392 external_mini_cluster.cc:1428] Setting key 7480e7813e30d6b1ec274fb64a93d9ba
I20260925 16:31:35.032691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:31:35.043728 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.031430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34627 (local address 127.0.98.62:40621)
0925 16:31:35.031622 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.031628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.033253 (+ 1625us) server_negotiation.cc:407] Connection header received
0925 16:31:35.036071 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.036076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.036176 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.036339 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:35.038020 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.038953 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.040621 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.041032 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.041192 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.042959 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:35.042964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:35.042968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:35.042991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:35.043028 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:35.043041 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:35.043146 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.043435 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.043541 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:31:35.045138 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.031969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.1:34627)
0925 16:31:35.033067 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.033129 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.035546 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.036678 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.036693 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.036844 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:35.037707 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.037730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.039235 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.039239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.040422 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.040437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.041282 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.041286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.041487 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.042512 (+ 1025us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:35.042514 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:35.042576 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:35.042705 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:35.043217 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:35.043228 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:35.043234 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.043390 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.043785 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":186,"threads_started":1}
I20260925 16:31:35.047223 20311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260925 16:31:35.047581 20311 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:35.048408 20311 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260925 16:31:35.050684 20122 ts_manager.cc:194] Registered new tserver with Master: 01d62f29ae4e44499f76bd679ab84f66 (127.0.98.1:34519)
I20260925 16:31:35.052491 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34627
W20260925 16:31:35.170214 20314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:35.170534 20314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:35.170580 20314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:35.175208 20314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:35.175300 20314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:35.175364 20314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 16:31:35.180850 20314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:35.182153 20314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:35.184351 20314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:35.196575 20319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:35.196666 20320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:35.196575 20322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:35.196835 20314 server_base.cc:1034] running on GCE node
I20260925 16:31:35.197892 20314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:35.199064 20314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:35.200417 20314 hybrid_clock.cc:648] HybridClock initialized: now 1790353895200386 us; error 60 us; skew 500 ppm
I20260925 16:31:35.203481 20314 webserver.cc:460] Webserver started at http://127.0.98.2:36643/ using document root <none> and password file <none>
I20260925 16:31:35.204203 20314 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:35.204281 20314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:35.204464 20314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:35.206885 20314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data/instance:
uuid: "868a996a714f4318b3b86f2569d316f1"
format_stamp: "Formatted at 2026-09-25 16:31:35 on dist-test-slave-31kt"
server_key: "b28b97bc20aeb4df7ca4060a42c5f526"
server_key_iv: "9de6df3a3ca6caa7afc3e968523749a2"
server_key_version: "encryptionkey@0"
I20260925 16:31:35.207513 20314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "868a996a714f4318b3b86f2569d316f1"
format_stamp: "Formatted at 2026-09-25 16:31:35 on dist-test-slave-31kt"
server_key: "b28b97bc20aeb4df7ca4060a42c5f526"
server_key_iv: "9de6df3a3ca6caa7afc3e968523749a2"
server_key_version: "encryptionkey@0"
I20260925 16:31:35.212208 20314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 16:31:35.215608 20328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.217371 20314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 16:31:35.217556 20314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal
uuid: "868a996a714f4318b3b86f2569d316f1"
format_stamp: "Formatted at 2026-09-25 16:31:35 on dist-test-slave-31kt"
server_key: "b28b97bc20aeb4df7ca4060a42c5f526"
server_key_iv: "9de6df3a3ca6caa7afc3e968523749a2"
server_key_version: "encryptionkey@0"
I20260925 16:31:35.217694 20314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:35.245316 20314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:35.246337 20314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:35.246553 20314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:35.247314 20314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:35.248562 20314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:35.248617 20314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.248692 20314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:35.248716 20314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.260169 20314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33987
I20260925 16:31:35.260221 20441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33987 every 8 connection(s)
I20260925 16:31:35.261399 20314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/data/info.pb
I20260925 16:31:35.263103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20314
I20260925 16:31:35.263288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal/instance
I20260925 16:31:35.263633 392 external_mini_cluster.cc:1428] Setting key 98a1bd960a849ef5568e2c2068efdf0c
I20260925 16:31:35.266144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 16:31:35.274597 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.263800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55193 (local address 127.0.98.62:40621)
0925 16:31:35.263938 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.263943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.265728 (+ 1785us) server_negotiation.cc:407] Connection header received
0925 16:31:35.267541 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.267546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.267626 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.267713 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:35.269001 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.269868 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.271655 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.272066 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.272247 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.273695 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:35.273699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:35.273703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:35.273727 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:35.273798 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:35.273814 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:35.273915 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.274257 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.274369 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 16:31:35.276051 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.264510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.2:55193)
0925 16:31:35.265505 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.265575 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.267236 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.267966 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.267976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.268102 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:35.268806 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.268822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.270135 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.270140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.271406 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.271431 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.272326 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.272330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.272538 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.273283 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:35.273285 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:35.273353 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:35.273450 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:35.274015 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:35.274033 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:35.274042 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.274239 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.274695 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":153,"threads_started":1}
I20260925 16:31:35.277915 20442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260925 16:31:35.278266 20442 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:35.279045 20442 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260925 16:31:35.281033 20122 ts_manager.cc:194] Registered new tserver with Master: 868a996a714f4318b3b86f2569d316f1 (127.0.98.2:33987)
I20260925 16:31:35.281785 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55193
W20260925 16:31:35.410401 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:35.410842 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:35.410909 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:35.415771 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:35.415872 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:35.415968 20445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 16:31:35.421952 20445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:35.423703 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:35.425876 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:35.439263 20451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:35.439317 20450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:35.439424 20453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:35.439805 20445 server_base.cc:1034] running on GCE node
I20260925 16:31:35.440419 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:35.441534 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:35.442798 20445 hybrid_clock.cc:648] HybridClock initialized: now 1790353895442760 us; error 77 us; skew 500 ppm
I20260925 16:31:35.445704 20445 webserver.cc:460] Webserver started at http://127.0.98.3:40391/ using document root <none> and password file <none>
I20260925 16:31:35.446566 20445 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:35.446669 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:35.446935 20445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:35.450048 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1f3163560c943ac94d863443e84f832"
format_stamp: "Formatted at 2026-09-25 16:31:35 on dist-test-slave-31kt"
server_key: "9b704e9bdf6c88f384fb4112b1178f12"
server_key_iv: "eaf19833e0892f526a095e7821b40a32"
server_key_version: "encryptionkey@0"
I20260925 16:31:35.450891 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1f3163560c943ac94d863443e84f832"
format_stamp: "Formatted at 2026-09-25 16:31:35 on dist-test-slave-31kt"
server_key: "9b704e9bdf6c88f384fb4112b1178f12"
server_key_iv: "eaf19833e0892f526a095e7821b40a32"
server_key_version: "encryptionkey@0"
I20260925 16:31:35.456533 20445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:35.460781 20459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.463209 20445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:31:35.463407 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal
uuid: "c1f3163560c943ac94d863443e84f832"
format_stamp: "Formatted at 2026-09-25 16:31:35 on dist-test-slave-31kt"
server_key: "9b704e9bdf6c88f384fb4112b1178f12"
server_key_iv: "eaf19833e0892f526a095e7821b40a32"
server_key_version: "encryptionkey@0"
I20260925 16:31:35.463580 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:35.482738 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:35.483625 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:35.483870 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:35.484995 20445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 16:31:35.486729 20445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 16:31:35.486804 20445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.486855 20445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 16:31:35.486873 20445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:35.498100 20445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45409
I20260925 16:31:35.498315 20572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45409 every 8 connection(s)
I20260925 16:31:35.499334 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/data/info.pb
I20260925 16:31:35.506498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20445
I20260925 16:31:35.506685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal/instance
I20260925 16:31:35.507011 392 external_mini_cluster.cc:1428] Setting key b15a64b1f546a2d9aed16b389b3da538
I20260925 16:31:35.512075 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.501836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59549 (local address 127.0.98.62:40621)
0925 16:31:35.502060 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.502064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.503512 (+ 1448us) server_negotiation.cc:407] Connection header received
0925 16:31:35.505212 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.505216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.505293 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.505399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:35.506720 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.507344 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.509263 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.509639 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.509808 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.511389 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:35.511393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:35.511396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:35.511413 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:35.511440 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:35.511450 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:35.511527 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.511812 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.511915 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 16:31:35.513304 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.502324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.3:59549)
0925 16:31:35.503308 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.503368 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.504867 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.505676 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.505685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.505861 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:31:35.506514 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.506532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.507545 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.507549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.509082 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.509100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.509924 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.509930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.510153 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.510994 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 16:31:35.510996 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:31:35.511060 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 16:31:35.511168 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:31:35.511536 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:31:35.511542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:31:35.511547 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.511672 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.512057 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":151,"threads_started":1}
I20260925 16:31:35.515059 20573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260925 16:31:35.515364 20573 heartbeater.cc:464] Registering TS with master...
I20260925 16:31:35.516103 20573 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260925 16:31:35.517802 20122 ts_manager.cc:194] Registered new tserver with Master: c1f3163560c943ac94d863443e84f832 (127.0.98.3:45409)
I20260925 16:31:35.518482 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59549
I20260925 16:31:35.522692 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 16:31:35.530153 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.525459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.0.98.62:40621)
0925 16:31:35.525651 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.525657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.525945 (+ 288us) server_negotiation.cc:407] Connection header received
0925 16:31:35.526028 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.526031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.526088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.526171 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:35.526535 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.527146 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.528154 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.528419 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.528522 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.529525 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:35.529528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:35.529531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:35.529549 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:35.529580 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:35.529589 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:35.529667 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.529847 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.529954 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:31:35.533828 20122 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 16:31:35.538326 20122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 16:31:35.557278 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.550099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.98.1:34519)
0925 16:31:35.551493 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.551502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.551955 (+ 453us) server_negotiation.cc:407] Connection header received
0925 16:31:35.552048 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.552053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.552305 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.552497 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:35.552748 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.554052 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.555702 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.556427 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.556556 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.556824 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.556926 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.557076 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":145,"threads_started":1}
I20260925 16:31:35.557271 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.549793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34519 (local address 127.0.0.1:46432)
0925 16:31:35.551709 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.551725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.551844 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.552459 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.552462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.552484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:35.552613 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.552622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.554188 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.554193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.555524 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.555538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.556579 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.556582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.556716 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.556759 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.556887 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.557070 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"mutex_wait_us":693,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260925 16:31:35.557492 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.550687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.0.98.3:45409)
0925 16:31:35.551575 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.551589 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.551622 (+ 33us) server_negotiation.cc:407] Connection header received
0925 16:31:35.551720 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.551725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.551999 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.552190 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:35.552608 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.554015 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.555913 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.556729 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.556941 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.557142 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.557199 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.557311 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":175,"threads_started":1}
I20260925 16:31:35.557487 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.550102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45409 (local address 127.0.0.1:54480)
0925 16:31:35.550966 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.551039 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.551209 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.552201 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.552207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.552234 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:35.552426 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.552436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.554188 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.554193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.555759 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.555770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.556888 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.556892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.557029 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.557059 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.557128 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.557246 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":141,"spinlock_wait_cycles":3328,"thread_start_us":169,"threads_started":1}
I20260925 16:31:35.558097 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.549793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33987 (local address 127.0.0.1:41990)
0925 16:31:35.550966 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.551039 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.551234 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.552406 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.552409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.552431 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:35.552631 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.552638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.553748 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.553781 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.555340 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.555352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.556888 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.556892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.557652 (+ 760us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.557679 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.557797 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.557895 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":89,"spinlock_wait_cycles":9088,"thread_start_us":104,"threads_started":1}
I20260925 16:31:35.558173 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.550304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.98.2:33987)
0925 16:31:35.551581 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.551593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.551622 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:31:35.551719 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.551725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.551999 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.552226 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:35.552783 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.553624 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.555529 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.556431 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.556567 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.556816 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.557847 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.557972 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":113,"threads_started":1}
I20260925 16:31:35.562109 20245 tablet_service.cc:1467] Processing CreateTablet for tablet 70aa9388d7cf49a6b999a36ea7724a34 (DEFAULT_TABLE table=test-table [id=4ab4a6ca1e9d44c3a31762ab90195150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:35.562477 20376 tablet_service.cc:1467] Processing CreateTablet for tablet 70aa9388d7cf49a6b999a36ea7724a34 (DEFAULT_TABLE table=test-table [id=4ab4a6ca1e9d44c3a31762ab90195150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:35.562479 20507 tablet_service.cc:1467] Processing CreateTablet for tablet 70aa9388d7cf49a6b999a36ea7724a34 (DEFAULT_TABLE table=test-table [id=4ab4a6ca1e9d44c3a31762ab90195150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 16:31:35.563411 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70aa9388d7cf49a6b999a36ea7724a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:35.563875 20376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70aa9388d7cf49a6b999a36ea7724a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:35.563875 20507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70aa9388d7cf49a6b999a36ea7724a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:35.571568 20590 tablet_bootstrap.cc:492] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66: Bootstrap starting.
I20260925 16:31:35.573935 20591 tablet_bootstrap.cc:492] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Bootstrap starting.
I20260925 16:31:35.575018 20590 tablet_bootstrap.cc:654] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:35.576078 20590 log.cc:824] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:35.576623 20591 tablet_bootstrap.cc:654] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:35.577111 20592 tablet_bootstrap.cc:492] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Bootstrap starting.
I20260925 16:31:35.577530 20591 log.cc:824] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:35.579141 20590 tablet_bootstrap.cc:492] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66: No bootstrap required, opened a new log
I20260925 16:31:35.579419 20590 ts_tablet_manager.cc:1397] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 16:31:35.579582 20592 tablet_bootstrap.cc:654] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:35.580300 20591 tablet_bootstrap.cc:492] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: No bootstrap required, opened a new log
I20260925 16:31:35.580468 20592 log.cc:824] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:35.580561 20591 ts_tablet_manager.cc:1397] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 16:31:35.583225 20592 tablet_bootstrap.cc:492] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: No bootstrap required, opened a new log
I20260925 16:31:35.583473 20592 ts_tablet_manager.cc:1397] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 16:31:35.584630 20590 raft_consensus.cc:348] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.584991 20590 raft_consensus.cc:374] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:35.585049 20590 raft_consensus.cc:729] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d62f29ae4e44499f76bd679ab84f66, State: Initialized, Role: FOLLOWER
I20260925 16:31:35.585219 20591 raft_consensus.cc:348] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.585601 20591 raft_consensus.cc:374] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:35.585666 20591 raft_consensus.cc:729] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868a996a714f4318b3b86f2569d316f1, State: Initialized, Role: FOLLOWER
I20260925 16:31:35.585644 20590 consensus_queue.cc:260] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.586251 20591 consensus_queue.cc:260] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.586782 20311 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260925 16:31:35.587183 20590 ts_tablet_manager.cc:1428] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 16:31:35.587502 20591 ts_tablet_manager.cc:1428] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:31:35.587919 20442 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260925 16:31:35.588094 20592 raft_consensus.cc:348] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.588430 20592 raft_consensus.cc:374] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:35.588486 20592 raft_consensus.cc:729] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f3163560c943ac94d863443e84f832, State: Initialized, Role: FOLLOWER
I20260925 16:31:35.589071 20592 consensus_queue.cc:260] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.589972 20573 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260925 16:31:35.590346 20592 ts_tablet_manager.cc:1428] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 16:31:35.602643 20597 raft_consensus.cc:482] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 16:31:35.602893 20597 raft_consensus.cc:504] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.604183 20597 leader_election.cc:288] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d62f29ae4e44499f76bd679ab84f66 (127.0.98.1:34519), c1f3163560c943ac94d863443e84f832 (127.0.98.3:45409)
I20260925 16:31:35.608049 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.604685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56543 (local address 127.0.98.3:45409)
0925 16:31:35.604858 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.604863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.604880 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:31:35.604951 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.604954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.605003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.605071 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:35.605344 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.605936 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.607085 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.607493 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.607587 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.607654 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.607831 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.607907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 16:31:35.608048 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.604523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45409 (local address 127.0.98.2:56543)
0925 16:31:35.604732 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.604748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.604849 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.605098 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.605101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.605119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:35.605242 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.605249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.606044 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.606048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.606959 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.606971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.607612 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.607616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.607721 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.607740 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.607805 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.607898 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 16:31:35.608704 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.604668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37401 (local address 127.0.98.1:34519)
0925 16:31:35.604894 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.604900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.605292 (+ 392us) server_negotiation.cc:407] Connection header received
0925 16:31:35.605477 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.605481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.605532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.605595 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 16:31:35.605929 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.606497 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.607637 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.608063 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.608158 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.608231 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.608464 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.608555 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 16:31:35.608880 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aa9388d7cf49a6b999a36ea7724a34" candidate_uuid: "868a996a714f4318b3b86f2569d316f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f3163560c943ac94d863443e84f832" is_pre_election: true
I20260925 16:31:35.609131 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.604589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34519 (local address 127.0.98.2:37401)
0925 16:31:35.605196 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.605208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 16:31:35.605335 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:31:35.605627 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:31:35.605630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:31:35.605646 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 16:31:35.605827 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.605835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.606605 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.606609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.607509 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:31:35.607518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.608207 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.608211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:31:35.608320 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.608346 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 16:31:35.608887 (+ 541us) client_negotiation.cc:240] Negotiation successful
0925 16:31:35.608976 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":43,"thread_start_us":184,"threads_started":1}
I20260925 16:31:35.609243 20527 raft_consensus.cc:2454] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868a996a714f4318b3b86f2569d316f1 in term 0.
I20260925 16:31:35.609856 20329 leader_election.cc:302] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868a996a714f4318b3b86f2569d316f1, c1f3163560c943ac94d863443e84f832; no voters:
I20260925 16:31:35.609882 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aa9388d7cf49a6b999a36ea7724a34" candidate_uuid: "868a996a714f4318b3b86f2569d316f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d62f29ae4e44499f76bd679ab84f66" is_pre_election: true
I20260925 16:31:35.610195 20597 raft_consensus.cc:2781] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 16:31:35.610266 20265 raft_consensus.cc:2454] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868a996a714f4318b3b86f2569d316f1 in term 0.
I20260925 16:31:35.610330 20597 raft_consensus.cc:482] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 16:31:35.610373 20597 raft_consensus.cc:3037] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:35.611726 20597 raft_consensus.cc:504] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.612157 20597 leader_election.cc:288] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [CANDIDATE]: Term 1 election: Requested vote from peers 01d62f29ae4e44499f76bd679ab84f66 (127.0.98.1:34519), c1f3163560c943ac94d863443e84f832 (127.0.98.3:45409)
I20260925 16:31:35.612711 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aa9388d7cf49a6b999a36ea7724a34" candidate_uuid: "868a996a714f4318b3b86f2569d316f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f3163560c943ac94d863443e84f832"
I20260925 16:31:35.612800 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aa9388d7cf49a6b999a36ea7724a34" candidate_uuid: "868a996a714f4318b3b86f2569d316f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d62f29ae4e44499f76bd679ab84f66"
I20260925 16:31:35.612898 20527 raft_consensus.cc:3037] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:35.612934 20265 raft_consensus.cc:3037] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:35.614260 20527 raft_consensus.cc:2454] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868a996a714f4318b3b86f2569d316f1 in term 1.
I20260925 16:31:35.614305 20265 raft_consensus.cc:2454] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868a996a714f4318b3b86f2569d316f1 in term 1.
I20260925 16:31:35.614728 20329 leader_election.cc:302] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868a996a714f4318b3b86f2569d316f1, c1f3163560c943ac94d863443e84f832; no voters:
I20260925 16:31:35.615127 20597 raft_consensus.cc:2781] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:35.615486 20597 raft_consensus.cc:686] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 1 LEADER]: Becoming Leader. State: Replica: 868a996a714f4318b3b86f2569d316f1, State: Running, Role: LEADER
I20260925 16:31:35.616097 20597 consensus_queue.cc:237] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } }
I20260925 16:31:35.621081 20119 catalog_manager.cc:5495] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 868a996a714f4318b3b86f2569d316f1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "868a996a714f4318b3b86f2569d316f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868a996a714f4318b3b86f2569d316f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } health_report { overall_health: UNKNOWN } } }
I20260925 16:31:35.644253 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.639994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.0.98.2:33987)
0925 16:31:35.640133 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:35.640136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:35.640155 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:31:35.640264 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:35.640267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:35.640330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:35.640427 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:31:35.640676 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.641241 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.642389 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:35.642651 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:35.642756 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:35.643193 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:31:35.643779 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:31:35.643911 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:35.644036 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 16:31:35.644113 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:31:35.652124 20527 raft_consensus.cc:1261] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 1 FOLLOWER]: Refusing update from remote peer 868a996a714f4318b3b86f2569d316f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:35.652145 20265 raft_consensus.cc:1261] T 70aa9388d7cf49a6b999a36ea7724a34 P 01d62f29ae4e44499f76bd679ab84f66 [term 1 FOLLOWER]: Refusing update from remote peer 868a996a714f4318b3b86f2569d316f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:31:35.653193 20600 consensus_queue.cc:1035] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f3163560c943ac94d863443e84f832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:35.653432 20597 consensus_queue.cc:1035] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d62f29ae4e44499f76bd679ab84f66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:31:35.661962 20603 mvcc.cc:205] Tried to move back new op lower bound from 7333289556581187584 to 7333289556446986240. Current Snapshot: MvccSnapshot[applied={T|T < 7333289556581187584}]
I20260925 16:31:35.662638 20605 mvcc.cc:205] Tried to move back new op lower bound from 7333289556581187584 to 7333289556446986240. Current Snapshot: MvccSnapshot[applied={T|T < 7333289556581187584}]
I20260925 16:31:35.663813 20604 mvcc.cc:205] Tried to move back new op lower bound from 7333289556581187584 to 7333289556446986240. Current Snapshot: MvccSnapshot[applied={T|T < 7333289556581187584}]
I20260925 16:31:35.676378 20119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:35.676692 20119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 16:31:35.680480 20119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fc6a44bedd44a45a00fddf2c37e0994: Sending DeleteTablet for 3 replicas of tablet 70aa9388d7cf49a6b999a36ea7724a34
I20260925 16:31:35.681576 20245 tablet_service.cc:1514] Processing DeleteTablet for tablet 70aa9388d7cf49a6b999a36ea7724a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:35 UTC) from {username='slave'} at 127.0.0.1:46432
I20260925 16:31:35.681882 20507 tablet_service.cc:1514] Processing DeleteTablet for tablet 70aa9388d7cf49a6b999a36ea7724a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:35 UTC) from {username='slave'} at 127.0.0.1:54480
I20260925 16:31:35.682245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20183
I20260925 16:31:35.685397 20376 tablet_service.cc:1514] Processing DeleteTablet for tablet 70aa9388d7cf49a6b999a36ea7724a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 16:31:35 UTC) from {username='slave'} at 127.0.0.1:41990
I20260925 16:31:35.685493 20613 tablet_replica.cc:330] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: stopping tablet replica
I20260925 16:31:35.685928 20613 raft_consensus.cc:2229] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 16:31:35.686281 20613 raft_consensus.cc:2258] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:35.686715 20614 tablet_replica.cc:330] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: stopping tablet replica
I20260925 16:31:35.687218 20614 raft_consensus.cc:2229] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 16:31:35.687973 20614 raft_consensus.cc:2258] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 16:31:35.689139 20613 ts_tablet_manager.cc:1905] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 16:31:35.690165 20614 ts_tablet_manager.cc:1905] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 16:31:35.692901 20109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34519 (error 108)
I20260925 16:31:35.693890 20614 ts_tablet_manager.cc:1918] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:35.693890 20613 ts_tablet_manager.cc:1918] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 16:31:35.694051 20614 log.cc:1196] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-1/wal/wals/70aa9388d7cf49a6b999a36ea7724a34
I20260925 16:31:35.694283 20613 log.cc:1196] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Deleting WAL directory at /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790353612433705-392-0/minicluster-data/ts-2/wal/wals/70aa9388d7cf49a6b999a36ea7724a34
I20260925 16:31:35.694427 20614 ts_tablet_manager.cc:1939] T 70aa9388d7cf49a6b999a36ea7724a34 P 868a996a714f4318b3b86f2569d316f1: Deleting consensus metadata
I20260925 16:31:35.694928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20314
I20260925 16:31:35.694959 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.694558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34519 (local address 127.0.0.1:46444)
0925 16:31:35.694734 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.694834 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 16:31:35.695308 20109 catalog_manager.cc:4584] TS 01d62f29ae4e44499f76bd679ab84f66 (127.0.98.1:34519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70aa9388d7cf49a6b999a36ea7724a34: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34519: connect: Connection refused (error 111)
I20260925 16:31:35.696674 20613 ts_tablet_manager.cc:1939] T 70aa9388d7cf49a6b999a36ea7724a34 P c1f3163560c943ac94d863443e84f832: Deleting consensus metadata
I20260925 16:31:35.698813 20107 catalog_manager.cc:4841] TS c1f3163560c943ac94d863443e84f832 (127.0.98.3:45409): tablet 70aa9388d7cf49a6b999a36ea7724a34 (table test-table [id=4ab4a6ca1e9d44c3a31762ab90195150]) successfully deleted
I20260925 16:31:35.703154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20445
I20260925 16:31:35.704798 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:35.704346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33987 (local address 127.0.0.1:42008)
0925 16:31:35.704599 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:31:35.704664 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 16:31:35.709924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20092
2026-09-25T16:31:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1227 ms)
[----------] 6 tests from EncryptionPolicyTest (11871 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:35 dist-test-slave-31kt krb5kdc[20618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:35 dist-test-slave-31kt krb5kdc[20618](info): set up 2 sockets
Sep 25 16:31:35 dist-test-slave-31kt krb5kdc[20618](info): commencing operation
krb5kdc: starting...
W20260925 16:31:36.778168 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:36 dist-test-slave-31kt krb5kdc[20618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:36.918023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:37.050354 20631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:37.050796 20631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:37.050863 20631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:37.057193 20631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:37.057332 20631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:37.057355 20631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:37.057374 20631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:37.057389 20631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:37.064740 20631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:37.066391 20631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:37.067947 20631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:37.079334 20636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:37.079362 20637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:37.079334 20639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:37.080323 20631 server_base.cc:1034] running on GCE node
I20260925 16:31:37.081080 20631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:37.082719 20631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:37.083987 20631 hybrid_clock.cc:648] HybridClock initialized: now 1790353897083953 us; error 74 us; skew 500 ppm
Sep 25 16:31:37 dist-test-slave-31kt krb5kdc[20618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:37.094781 20631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:37.096390 20631 webserver.cc:460] Webserver started at http://127.0.0.1:42079/ using document root <none> and password file <none>
I20260925 16:31:37.097411 20631 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:37.097530 20631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:37.097817 20631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:37.100447 20631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "0ae585db247a4e9ab90fb7561d231543"
format_stamp: "Formatted at 2026-09-25 16:31:37 on dist-test-slave-31kt"
server_key: "c45b352fcc3fcdcee5f15f6d4a8772d5"
server_key_iv: "f5fa460a834015063464da05c9139550"
server_key_version: "encryptionkey@0"
I20260925 16:31:37.101392 20631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ae585db247a4e9ab90fb7561d231543"
format_stamp: "Formatted at 2026-09-25 16:31:37 on dist-test-slave-31kt"
server_key: "c45b352fcc3fcdcee5f15f6d4a8772d5"
server_key_iv: "f5fa460a834015063464da05c9139550"
server_key_version: "encryptionkey@0"
I20260925 16:31:37.106848 20631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:31:37.110409 20646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:37.112416 20631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:37.112608 20631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "0ae585db247a4e9ab90fb7561d231543"
format_stamp: "Formatted at 2026-09-25 16:31:37 on dist-test-slave-31kt"
server_key: "c45b352fcc3fcdcee5f15f6d4a8772d5"
server_key_iv: "f5fa460a834015063464da05c9139550"
server_key_version: "encryptionkey@0"
I20260925 16:31:37.112769 20631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:37.137745 20631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:37.142028 20631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:37.142269 20631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:37.150851 20631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41207
I20260925 16:31:37.150954 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41207 every 8 connection(s)
I20260925 16:31:37.152046 20631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:37.156212 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:37.157651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20631
I20260925 16:31:37.157846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:37.158209 392 external_mini_cluster.cc:1428] Setting key ee711f05e615e7e4cfdb754760ad58ff
I20260925 16:31:37.167009 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543: Bootstrap starting.
Sep 25 16:31:37 dist-test-slave-31kt krb5kdc[20618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:31:37.170651 20699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:37.171828 20699 log.cc:824] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:37.175220 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543: No bootstrap required, opened a new log
I20260925 16:31:37.176836 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:37.160098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60241 (local address 127.0.0.1:41207)
0925 16:31:37.160850 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:37.160871 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:37.160931 (+ 60us) server_negotiation.cc:407] Connection header received
0925 16:31:37.161968 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:37.161996 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:37.162451 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:37.162870 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:37.163846 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:37.164960 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:37.165955 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:37.166535 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:37.166717 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:37.170235 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:37.170242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:37.170264 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:37.170308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:37.172588 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:37.173537 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:37.173544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:37.173550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:37.173712 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:37.174158 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:37.174161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:37.174162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:37.174685 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:37.174883 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:37.175385 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 16:31:37.175925 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":179,"threads_started":1}
I20260925 16:31:37.180264 20699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae585db247a4e9ab90fb7561d231543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260925 16:31:37.180640 20699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:37.180722 20699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae585db247a4e9ab90fb7561d231543, State: Initialized, Role: FOLLOWER
I20260925 16:31:37.180882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:37.181327 20699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae585db247a4e9ab90fb7561d231543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260925 16:31:37.181504 20699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:37.181574 20699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:37.181684 20699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:37.183066 20699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae585db247a4e9ab90fb7561d231543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260925 16:31:37.183626 20699 leader_election.cc:302] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ae585db247a4e9ab90fb7561d231543; no voters:
I20260925 16:31:37.184108 20699 leader_election.cc:288] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:37.184288 20709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:37.184609 20709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [term 1 LEADER]: Becoming Leader. State: Replica: 0ae585db247a4e9ab90fb7561d231543, State: Running, Role: LEADER
I20260925 16:31:37.185074 20709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae585db247a4e9ab90fb7561d231543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260925 16:31:37.186095 20699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:37.186796 20710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ae585db247a4e9ab90fb7561d231543. Latest consensus state: current_term: 1 leader_uuid: "0ae585db247a4e9ab90fb7561d231543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae585db247a4e9ab90fb7561d231543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } } }
I20260925 16:31:37.186995 20710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:37.187325 20711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ae585db247a4e9ab90fb7561d231543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae585db247a4e9ab90fb7561d231543" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } } }
I20260925 16:31:37.187523 20711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543 [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:37.192497 20724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:37.192600 20724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:37.192695 20715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:37.193804 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:37.183224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50657 (local address 127.0.0.1:41207)
0925 16:31:37.183393 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:37.183398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:37.183709 (+ 311us) server_negotiation.cc:407] Connection header received
0925 16:31:37.183829 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:37.183833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:37.183901 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:37.183994 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:37.184959 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:37.185716 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:37.187020 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:37.187399 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:37.187478 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:37.189565 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:37.189569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:37.189572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:37.189601 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:37.191265 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:37.192076 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:37.192080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:37.192082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:37.192191 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:37.192591 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:37.192595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:37.192596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:37.192833 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:37.192978 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:37.193446 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 16:31:37.193589 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:31:37.193907 20715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:37.201655 20715 catalog_manager.cc:1347] Generated new cluster ID: c511dfb8b2ce4ec2899d1449b65e2a58
I20260925 16:31:37.201820 20715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:37.217451 20715 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:37.219287 20715 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:37.233006 20715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ae585db247a4e9ab90fb7561d231543: Generated new TSK 0
I20260925 16:31:37.234395 20715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:37.249354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20631
W20260925 16:31:37.256428 20616 connection.cc:535] client connection to 127.0.0.1:41207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41207): Connection reset by peer (error 104)
2026-09-25T16:31:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:37 dist-test-slave-31kt krb5kdc[20730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:37 dist-test-slave-31kt krb5kdc[20730](info): set up 2 sockets
Sep 25 16:31:37 dist-test-slave-31kt krb5kdc[20730](info): commencing operation
krb5kdc: starting...
W20260925 16:31:38.222762 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:38 dist-test-slave-31kt krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:38.374497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:38.516342 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:38.516784 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:38.516855 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:38.521490 20743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:38.521623 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:38.521651 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:38.521672 20743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:38.521693 20743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:38.528797 20743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:38.530506 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:38.532034 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:38.543387 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:38.543409 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:38.543720 20743 server_base.cc:1034] running on GCE node
W20260925 16:31:38.543417 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:38.544730 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:38.546510 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:38.547801 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790353898547761 us; error 65 us; skew 500 ppm
Sep 25 16:31:38 dist-test-slave-31kt krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:38.552244 20743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:38.553702 20743 webserver.cc:460] Webserver started at http://127.0.0.1:41637/ using document root <none> and password file <none>
I20260925 16:31:38.554541 20743 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:38.554620 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:38.554867 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:38.557696 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "5325c7118cf44be7ae54c3efae183ed1"
format_stamp: "Formatted at 2026-09-25 16:31:38 on dist-test-slave-31kt"
server_key: "ee21f61b6a6f22d3035c425252b22966"
server_key_iv: "bcedad4e4766f970ec466467330be98a"
server_key_version: "encryptionkey@0"
I20260925 16:31:38.558475 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5325c7118cf44be7ae54c3efae183ed1"
format_stamp: "Formatted at 2026-09-25 16:31:38 on dist-test-slave-31kt"
server_key: "ee21f61b6a6f22d3035c425252b22966"
server_key_iv: "bcedad4e4766f970ec466467330be98a"
server_key_version: "encryptionkey@0"
I20260925 16:31:38.564219 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:38.568267 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:38.570500 20743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:31:38.570781 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "5325c7118cf44be7ae54c3efae183ed1"
format_stamp: "Formatted at 2026-09-25 16:31:38 on dist-test-slave-31kt"
server_key: "ee21f61b6a6f22d3035c425252b22966"
server_key_iv: "bcedad4e4766f970ec466467330be98a"
server_key_version: "encryptionkey@0"
I20260925 16:31:38.570991 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:38.590902 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:38.595340 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:38.595566 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:38.605520 20743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34777
I20260925 16:31:38.605581 20810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34777 every 8 connection(s)
I20260925 16:31:38.606962 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:38.610495 20811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:38.614049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20743
I20260925 16:31:38.614236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:38.614549 392 external_mini_cluster.cc:1428] Setting key c40bdc31404508f9297668787898034c
I20260925 16:31:38.617313 20811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1: Bootstrap starting.
I20260925 16:31:38.620210 20811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:38.621074 20811 log.cc:824] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:38.623965 20811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1: No bootstrap required, opened a new log
I20260925 16:31:38.626907 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:38.616423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36687 (local address 127.0.0.1:34777)
0925 16:31:38.617043 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:38.617063 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:38.617113 (+ 50us) server_negotiation.cc:407] Connection header received
0925 16:31:38.618093 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:38.618120 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:38.618601 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:38.619015 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:38.619664 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:38.620998 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:38.622392 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:38.623047 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:38.623214 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:38.624356 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:38.624364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:38.624388 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:38.625005 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:38.625096 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:38.625134 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:38.625301 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:38.625635 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 16:31:38.625835 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":192,"threads_started":1}
I20260925 16:31:38.628628 20811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c7118cf44be7ae54c3efae183ed1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260925 16:31:38.628962 20811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:38.629033 20811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5325c7118cf44be7ae54c3efae183ed1, State: Initialized, Role: FOLLOWER
I20260925 16:31:38.629536 20811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c7118cf44be7ae54c3efae183ed1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260925 16:31:38.629710 20811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:38.629802 20811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:38.629901 20811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:38.631078 20811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c7118cf44be7ae54c3efae183ed1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260925 16:31:38.631505 20811 leader_election.cc:302] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5325c7118cf44be7ae54c3efae183ed1; no voters:
I20260925 16:31:38.631608 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:38.631851 20811 leader_election.cc:288] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:38.632040 20816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:38.632398 20816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [term 1 LEADER]: Becoming Leader. State: Replica: 5325c7118cf44be7ae54c3efae183ed1, State: Running, Role: LEADER
I20260925 16:31:38.632915 20816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c7118cf44be7ae54c3efae183ed1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260925 16:31:38.633281 20811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:38.634783 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5325c7118cf44be7ae54c3efae183ed1. Latest consensus state: current_term: 1 leader_uuid: "5325c7118cf44be7ae54c3efae183ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c7118cf44be7ae54c3efae183ed1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } } }
I20260925 16:31:38.634953 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:38.635205 20817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5325c7118cf44be7ae54c3efae183ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c7118cf44be7ae54c3efae183ed1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } } }
I20260925 16:31:38.635355 20817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1 [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:38.639228 20835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:38.639361 20835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:38.639444 20831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:38.640616 20831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:38.641966 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:38.635779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45137 (local address 127.0.0.1:34777)
0925 16:31:38.635918 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:38.635923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:38.636576 (+ 653us) server_negotiation.cc:407] Connection header received
0925 16:31:38.636809 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:38.636812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:38.636867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:38.636925 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:38.637378 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:38.638295 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:38.639523 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:38.639843 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:38.639923 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:38.641267 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:38.641271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:38.641274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:38.641290 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:38.641322 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:38.641336 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:38.641421 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:38.641671 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 16:31:38.641717 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 16:31:38.648747 20831 catalog_manager.cc:1347] Generated new cluster ID: 15602f90abe749ecb364c8e5d9e82399
I20260925 16:31:38.648837 20831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:38.663170 20831 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:38.664321 20831 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:38.678809 20831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5325c7118cf44be7ae54c3efae183ed1: Generated new TSK 0
I20260925 16:31:38.680178 20831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:38.693543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20743
W20260925 16:31:38.699960 20728 connection.cc:535] client connection to 127.0.0.1:34777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34777): Connection reset by peer (error 104)
2026-09-25T16:31:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:38 dist-test-slave-31kt krb5kdc[20842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:38 dist-test-slave-31kt krb5kdc[20842](info): set up 2 sockets
Sep 25 16:31:38 dist-test-slave-31kt krb5kdc[20842](info): commencing operation
krb5kdc: starting...
W20260925 16:31:39.678442 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:39 dist-test-slave-31kt krb5kdc[20842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:39.821828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:39.958865 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:39.959278 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:39.959329 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:39.964169 20855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:39.964299 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:39.964320 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:39.964339 20855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:39.964353 20855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:39.971189 20855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:39.972818 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:39.974355 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:39.986393 20860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:39.986393 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:39.986639 20855 server_base.cc:1034] running on GCE node
W20260925 16:31:39.986465 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:39.987632 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:39.989084 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:39.992470 20855 hybrid_clock.cc:648] HybridClock initialized: now 1790353899992407 us; error 95 us; skew 500 ppm
Sep 25 16:31:39 dist-test-slave-31kt krb5kdc[20842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:39.997175 20855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:39.998808 20855 webserver.cc:460] Webserver started at http://127.0.0.1:34847/ using document root <none> and password file <none>
I20260925 16:31:39.999691 20855 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:39.999782 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:40.000046 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:40.003207 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "969fa3ab3f414407be7b32faab35fa33"
format_stamp: "Formatted at 2026-09-25 16:31:39 on dist-test-slave-31kt"
server_key: "afd94fd6cb02ad2ca00646157c0ff7e1"
server_key_iv: "82f4f4ea8d15d11e1b6d645e1ebdf800"
server_key_version: "encryptionkey@0"
I20260925 16:31:40.003933 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "969fa3ab3f414407be7b32faab35fa33"
format_stamp: "Formatted at 2026-09-25 16:31:39 on dist-test-slave-31kt"
server_key: "afd94fd6cb02ad2ca00646157c0ff7e1"
server_key_iv: "82f4f4ea8d15d11e1b6d645e1ebdf800"
server_key_version: "encryptionkey@0"
I20260925 16:31:40.009078 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:31:40.012951 20870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:40.015349 20855 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:40.015614 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "969fa3ab3f414407be7b32faab35fa33"
format_stamp: "Formatted at 2026-09-25 16:31:39 on dist-test-slave-31kt"
server_key: "afd94fd6cb02ad2ca00646157c0ff7e1"
server_key_iv: "82f4f4ea8d15d11e1b6d645e1ebdf800"
server_key_version: "encryptionkey@0"
I20260925 16:31:40.015810 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:40.023139 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:40.027271 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:40.027498 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:40.036120 20855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33901
I20260925 16:31:40.036167 20922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33901 every 8 connection(s)
I20260925 16:31:40.037292 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:40.040616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20855
I20260925 16:31:40.040767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:40.041111 392 external_mini_cluster.cc:1428] Setting key 85f365fce12887068a2c6c3f5625ddcb
I20260925 16:31:40.041388 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:40.048903 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:40.043130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46119 (local address 127.0.0.1:33901)
0925 16:31:40.043894 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:40.043916 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:40.043974 (+ 58us) server_negotiation.cc:407] Connection header received
0925 16:31:40.045212 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:40.045241 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:40.045631 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:40.046115 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:40.047138 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:40.047144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:40.047160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:40.047171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:40.047233 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:40.047255 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:40.047414 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:40.047489 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 16:31:40.047671 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":192,"threads_started":1}
I20260925 16:31:40.051693 20923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33: Bootstrap starting.
I20260925 16:31:40.053326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:40.055260 20923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:40.056502 20923 log.cc:824] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:40.058135 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:40.055776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57099 (local address 127.0.0.1:33901)
0925 16:31:40.055977 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:40.055983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:40.056191 (+ 208us) server_negotiation.cc:407] Connection header received
0925 16:31:40.056318 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:40.056322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:40.056382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:40.056516 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:40.057499 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:40.057503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:40.057505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:40.057518 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:40.057555 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:40.057567 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:40.057668 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:40.057891 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 16:31:40.057991 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 16:31:40.059831 20923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33: No bootstrap required, opened a new log
I20260925 16:31:40.064852 20923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969fa3ab3f414407be7b32faab35fa33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33901 } }
I20260925 16:31:40.065233 20923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:40.065271 20923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969fa3ab3f414407be7b32faab35fa33, State: Initialized, Role: FOLLOWER
I20260925 16:31:40.065909 20923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969fa3ab3f414407be7b32faab35fa33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33901 } }
I20260925 16:31:40.066080 20923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:40.066140 20923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:40.066263 20923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:40.067562 20923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969fa3ab3f414407be7b32faab35fa33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33901 } }
I20260925 16:31:40.068046 20923 leader_election.cc:302] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 969fa3ab3f414407be7b32faab35fa33; no voters:
I20260925 16:31:40.068472 20923 leader_election.cc:288] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:40.068763 20933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:40.069162 20933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [term 1 LEADER]: Becoming Leader. State: Replica: 969fa3ab3f414407be7b32faab35fa33, State: Running, Role: LEADER
I20260925 16:31:40.069638 20933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969fa3ab3f414407be7b32faab35fa33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33901 } }
I20260925 16:31:40.070292 20923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:40.071992 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "969fa3ab3f414407be7b32faab35fa33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969fa3ab3f414407be7b32faab35fa33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33901 } } }
I20260925 16:31:40.072042 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 969fa3ab3f414407be7b32faab35fa33. Latest consensus state: current_term: 1 leader_uuid: "969fa3ab3f414407be7b32faab35fa33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969fa3ab3f414407be7b32faab35fa33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33901 } } }
I20260925 16:31:40.072291 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:40.072324 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33 [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:40.077081 20948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:40.077219 20948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:40.077329 20947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:40.078449 20947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:40.086833 20947 catalog_manager.cc:1347] Generated new cluster ID: e09d2b2580504f4d8f8c73aa505691ef
I20260925 16:31:40.086973 20947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:40.127417 20947 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:40.127924 20947 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:40.142032 20947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 969fa3ab3f414407be7b32faab35fa33: Generated new TSK 0
I20260925 16:31:40.143082 20947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:40.147126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20855
2026-09-25T16:31:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:40 dist-test-slave-31kt krb5kdc[20954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:40 dist-test-slave-31kt krb5kdc[20954](info): set up 2 sockets
Sep 25 16:31:40 dist-test-slave-31kt krb5kdc[20954](info): commencing operation
krb5kdc: starting...
W20260925 16:31:41.126320 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:41 dist-test-slave-31kt krb5kdc[20954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:41.266784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:41.407155 20967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:41.407495 20967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:41.407543 20967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:41.411885 20967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:41.411981 20967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:41.412000 20967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:41.412016 20967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:41.412031 20967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:41.418238 20967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:41.419617 20967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:41.420826 20967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:41.434589 20972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:41.434645 20975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:41.434660 20973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:41.434798 20967 server_base.cc:1034] running on GCE node
I20260925 16:31:41.435927 20967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:41.437803 20967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:41.439213 20967 hybrid_clock.cc:648] HybridClock initialized: now 1790353901439117 us; error 80 us; skew 500 ppm
Sep 25 16:31:41 dist-test-slave-31kt krb5kdc[20954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:41.443465 20967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:41.445206 20967 webserver.cc:460] Webserver started at http://127.0.0.1:39927/ using document root <none> and password file <none>
I20260925 16:31:41.446143 20967 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:41.446254 20967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:41.446557 20967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:41.449573 20967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "a4d3e64960e34ac1b7bc5259db7a7a32"
format_stamp: "Formatted at 2026-09-25 16:31:41 on dist-test-slave-31kt"
server_key: "0402851019883b42b3932c8f4c18aa84"
server_key_iv: "3a6b24d10f84670876328c6ba5e73c8b"
server_key_version: "encryptionkey@0"
I20260925 16:31:41.450510 20967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4d3e64960e34ac1b7bc5259db7a7a32"
format_stamp: "Formatted at 2026-09-25 16:31:41 on dist-test-slave-31kt"
server_key: "0402851019883b42b3932c8f4c18aa84"
server_key_iv: "3a6b24d10f84670876328c6ba5e73c8b"
server_key_version: "encryptionkey@0"
I20260925 16:31:41.457242 20967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 16:31:41.461843 20982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:41.464335 20967 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:31:41.464614 20967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "a4d3e64960e34ac1b7bc5259db7a7a32"
format_stamp: "Formatted at 2026-09-25 16:31:41 on dist-test-slave-31kt"
server_key: "0402851019883b42b3932c8f4c18aa84"
server_key_iv: "3a6b24d10f84670876328c6ba5e73c8b"
server_key_version: "encryptionkey@0"
I20260925 16:31:41.464826 20967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:41.496147 20967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:41.501555 20967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:41.501894 20967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:41.512244 20967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42777
I20260925 16:31:41.512338 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42777 every 8 connection(s)
I20260925 16:31:41.513701 20967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:41.516841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 20967
I20260925 16:31:41.517091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:41.517511 392 external_mini_cluster.cc:1428] Setting key 2e28af3a33a2116899b906a5663280ae
I20260925 16:31:41.518445 21035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:41.527387 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32: Bootstrap starting.
Sep 25 16:31:41 dist-test-slave-31kt krb5kdc[20954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:31:41.530951 21035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:41.532023 21035 log.cc:824] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:41.534850 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32: No bootstrap required, opened a new log
I20260925 16:31:41.538198 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:41.519596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58629 (local address 127.0.0.1:42777)
0925 16:31:41.520235 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:41.520255 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:41.520302 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:31:41.521152 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:41.521177 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:41.521667 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:41.522162 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:41.523162 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:41.524391 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:41.525376 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:41.526235 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:41.526443 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:41.530422 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:41.530430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:41.530455 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:41.530510 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:41.533027 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:41.533848 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:41.533856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:41.533864 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:41.534064 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:41.534652 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:41.534656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:41.534657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:41.535487 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:41.535742 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:41.536304 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 16:31:41.536696 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":200,"threads_started":1}
I20260925 16:31:41.541278 21035 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260925 16:31:41.541638 21035 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:41.541695 21035 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d3e64960e34ac1b7bc5259db7a7a32, State: Initialized, Role: FOLLOWER
I20260925 16:31:41.542275 21035 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260925 16:31:41.542439 21035 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:41.542507 21035 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:41.542625 21035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:41.543531 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:41.543984 21035 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260925 16:31:41.544479 21035 leader_election.cc:302] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d3e64960e34ac1b7bc5259db7a7a32; no voters:
I20260925 16:31:41.544935 21035 leader_election.cc:288] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:41.545221 21040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:41.545620 21040 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [term 1 LEADER]: Becoming Leader. State: Replica: a4d3e64960e34ac1b7bc5259db7a7a32, State: Running, Role: LEADER
I20260925 16:31:41.546183 21040 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260925 16:31:41.546754 21035 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:41.548552 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } } }
I20260925 16:31:41.548774 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:41.549119 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d3e64960e34ac1b7bc5259db7a7a32. Latest consensus state: current_term: 1 leader_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d3e64960e34ac1b7bc5259db7a7a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } } }
I20260925 16:31:41.549206 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:41.549306 21059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:41.554025 21059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:41.556975 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:41.546507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57061 (local address 127.0.0.1:42777)
0925 16:31:41.546734 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:41.546740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:41.546908 (+ 168us) server_negotiation.cc:407] Connection header received
0925 16:31:41.547162 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:41.547167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:41.547253 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:41.547415 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:41.548434 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:41.549318 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:41.550732 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:41.551092 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:41.551205 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:41.553289 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:41.553293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:41.553296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:41.553330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:41.554967 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:41.555468 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:41.555471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:41.555473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:41.555561 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:41.555828 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:41.555831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:41.555833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:41.556136 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:41.556274 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:41.556597 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 16:31:41.556775 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 16:31:41.561903 21059 catalog_manager.cc:1347] Generated new cluster ID: 282ed3b9b29747b9b11b804e41d0c8b1
I20260925 16:31:41.562031 21059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:41.584398 21059 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:41.585922 21059 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:41.594880 21059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d3e64960e34ac1b7bc5259db7a7a32: Generated new TSK 0
I20260925 16:31:41.596019 21059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:41.611423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 20967
2026-09-25T16:31:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:41 dist-test-slave-31kt krb5kdc[21066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:41 dist-test-slave-31kt krb5kdc[21066](info): set up 2 sockets
Sep 25 16:31:41 dist-test-slave-31kt krb5kdc[21066](info): commencing operation
krb5kdc: starting...
W20260925 16:31:42.590389 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:42 dist-test-slave-31kt krb5kdc[21066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:42.729321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:42.867558 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:42.867947 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:42.868013 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:42.872505 21079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:42.872593 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:42.872622 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:42.872645 21079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:42.872668 21079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:42.878922 21079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:42.880425 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:42.881816 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:42.892663 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:42.892663 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:42.892663 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:42.892807 21079 server_base.cc:1034] running on GCE node
I20260925 16:31:42.893867 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:42.895419 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:42.896757 21079 hybrid_clock.cc:648] HybridClock initialized: now 1790353902896696 us; error 88 us; skew 500 ppm
Sep 25 16:31:42 dist-test-slave-31kt krb5kdc[21066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:42.905222 21079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:42.906834 21079 webserver.cc:460] Webserver started at http://127.0.0.1:43037/ using document root <none> and password file <none>
I20260925 16:31:42.907649 21079 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:42.907735 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:42.908002 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:42.910921 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "f1dbe25750344258add8a99dde3774f0"
format_stamp: "Formatted at 2026-09-25 16:31:42 on dist-test-slave-31kt"
server_key: "a832014c2b269aa95a81b0cfd941b043"
server_key_iv: "4b52b408541fe5eac8fb0cdb8a66adb8"
server_key_version: "encryptionkey@0"
I20260925 16:31:42.911644 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1dbe25750344258add8a99dde3774f0"
format_stamp: "Formatted at 2026-09-25 16:31:42 on dist-test-slave-31kt"
server_key: "a832014c2b269aa95a81b0cfd941b043"
server_key_iv: "4b52b408541fe5eac8fb0cdb8a66adb8"
server_key_version: "encryptionkey@0"
I20260925 16:31:42.916570 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:42.920089 21094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:42.922022 21079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:42.922216 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "f1dbe25750344258add8a99dde3774f0"
format_stamp: "Formatted at 2026-09-25 16:31:42 on dist-test-slave-31kt"
server_key: "a832014c2b269aa95a81b0cfd941b043"
server_key_iv: "4b52b408541fe5eac8fb0cdb8a66adb8"
server_key_version: "encryptionkey@0"
I20260925 16:31:42.922408 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:42.941700 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:42.945717 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:42.945997 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:42.954952 21079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40675
I20260925 16:31:42.954982 21146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40675 every 8 connection(s)
I20260925 16:31:42.956238 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:42.958453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21079
I20260925 16:31:42.958636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:42.958987 392 external_mini_cluster.cc:1428] Setting key 82182b66010cb08370ab9ae5f36b9a69
I20260925 16:31:42.960142 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:42.968123 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0: Bootstrap starting.
I20260925 16:31:42.970691 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:42.961020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51075 (local address 127.0.0.1:40675)
0925 16:31:42.961584 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:42.961602 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:42.961649 (+ 47us) server_negotiation.cc:407] Connection header received
0925 16:31:42.962503 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:42.962523 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:42.962975 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:42.963392 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:42.963876 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:42.965008 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:42.966390 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:42.966999 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:42.967167 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:42.968229 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:42.968236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:42.968295 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:42.968844 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:42.968921 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:42.968951 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:42.969037 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:42.969320 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 16:31:42.969627 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":150,"threads_started":1}
I20260925 16:31:42.970819 21147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:42.971644 21147 log.cc:824] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:42.974370 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0: No bootstrap required, opened a new log
I20260925 16:31:42.974574 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:42.978601 21147 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dbe25750344258add8a99dde3774f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40675 } }
I20260925 16:31:42.978894 21147 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:42.978946 21147 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1dbe25750344258add8a99dde3774f0, State: Initialized, Role: FOLLOWER
I20260925 16:31:42.979498 21147 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dbe25750344258add8a99dde3774f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40675 } }
I20260925 16:31:42.979689 21147 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:42.979751 21147 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:42.979856 21147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:42.981117 21147 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dbe25750344258add8a99dde3774f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40675 } }
I20260925 16:31:42.981563 21147 leader_election.cc:302] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1dbe25750344258add8a99dde3774f0; no voters:
I20260925 16:31:42.982023 21147 leader_election.cc:288] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:42.982127 21157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:42.982280 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:42.977125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48867 (local address 127.0.0.1:40675)
0925 16:31:42.977285 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:42.977291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:42.977594 (+ 303us) server_negotiation.cc:407] Connection header received
0925 16:31:42.977785 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:42.977802 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:42.977866 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:42.977948 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:42.978336 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:42.978968 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:42.980126 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:42.980415 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:42.980512 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:42.981471 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:42.981474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:42.981476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:42.981490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:42.981522 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:42.981533 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:42.981598 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:42.981840 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 16:31:42.981949 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 16:31:42.982363 21157 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [term 1 LEADER]: Becoming Leader. State: Replica: f1dbe25750344258add8a99dde3774f0, State: Running, Role: LEADER
I20260925 16:31:42.982717 21157 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dbe25750344258add8a99dde3774f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40675 } }
I20260925 16:31:42.983909 21147 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:42.984468 21157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1dbe25750344258add8a99dde3774f0. Latest consensus state: current_term: 1 leader_uuid: "f1dbe25750344258add8a99dde3774f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dbe25750344258add8a99dde3774f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40675 } } }
I20260925 16:31:42.984614 21157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:42.984865 21157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1dbe25750344258add8a99dde3774f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dbe25750344258add8a99dde3774f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40675 } } }
I20260925 16:31:42.984961 21157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:42.985249 21161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:42.989554 21161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:42.998142 21161 catalog_manager.cc:1347] Generated new cluster ID: 45ebee20cc0c4df2aadbfb9effe565a6
I20260925 16:31:42.998236 21161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:43.012131 21161 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:43.013262 21161 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:43.024469 21161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1dbe25750344258add8a99dde3774f0: Generated new TSK 0
I20260925 16:31:43.025333 21161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:43.031656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21079
2026-09-25T16:31:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:43 dist-test-slave-31kt krb5kdc[21178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:43 dist-test-slave-31kt krb5kdc[21178](info): set up 2 sockets
Sep 25 16:31:43 dist-test-slave-31kt krb5kdc[21178](info): commencing operation
krb5kdc: starting...
W20260925 16:31:44.022591 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:44 dist-test-slave-31kt krb5kdc[21178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:44.160769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:44.297914 21191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:44.298305 21191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:44.298354 21191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:44.303117 21191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:44.303236 21191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:44.303256 21191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:44.303274 21191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:44.303289 21191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:44.310122 21191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:44.311636 21191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:44.312965 21191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:44.323704 21196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:44.323693 21197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:44.323774 21191 server_base.cc:1034] running on GCE node
W20260925 16:31:44.323693 21199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:44.324749 21191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:44.326231 21191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:44.327468 21191 hybrid_clock.cc:648] HybridClock initialized: now 1790353904327444 us; error 86 us; skew 500 ppm
Sep 25 16:31:44 dist-test-slave-31kt krb5kdc[21178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:44.330746 21191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:44.331864 21191 webserver.cc:460] Webserver started at http://127.0.0.1:40527/ using document root <none> and password file <none>
I20260925 16:31:44.332520 21191 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:44.332588 21191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:44.332777 21191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:44.335328 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "824bbe4f87d741f38a6d01cf10ede490"
format_stamp: "Formatted at 2026-09-25 16:31:44 on dist-test-slave-31kt"
server_key: "d6f80e1d5bfd699faf94766015e9526d"
server_key_iv: "8826b6a60b4dbdc561d7b17f7cde88cc"
server_key_version: "encryptionkey@0"
I20260925 16:31:44.335960 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "824bbe4f87d741f38a6d01cf10ede490"
format_stamp: "Formatted at 2026-09-25 16:31:44 on dist-test-slave-31kt"
server_key: "d6f80e1d5bfd699faf94766015e9526d"
server_key_iv: "8826b6a60b4dbdc561d7b17f7cde88cc"
server_key_version: "encryptionkey@0"
I20260925 16:31:44.341231 21191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:31:44.345242 21206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:44.347477 21191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:44.347695 21191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "824bbe4f87d741f38a6d01cf10ede490"
format_stamp: "Formatted at 2026-09-25 16:31:44 on dist-test-slave-31kt"
server_key: "d6f80e1d5bfd699faf94766015e9526d"
server_key_iv: "8826b6a60b4dbdc561d7b17f7cde88cc"
server_key_version: "encryptionkey@0"
I20260925 16:31:44.347908 21191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:44.355553 21191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:44.360836 21191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:44.361038 21191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:44.370337 21191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46253
I20260925 16:31:44.370407 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46253 every 8 connection(s)
I20260925 16:31:44.371483 21191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:44.375712 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:44.379696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21191
I20260925 16:31:44.379911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:44.380331 392 external_mini_cluster.cc:1428] Setting key fcd2243771d743b585be5c4a3fc37847
I20260925 16:31:44.384950 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490: Bootstrap starting.
I20260925 16:31:44.388096 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:44.382422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51721 (local address 127.0.0.1:46253)
0925 16:31:44.382969 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:44.382987 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:44.383032 (+ 45us) server_negotiation.cc:407] Connection header received
0925 16:31:44.383941 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:44.383964 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:44.384371 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:44.384749 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:44.385999 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:44.386008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:44.386030 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:44.386049 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:44.386155 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:44.386190 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:44.386375 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:44.386629 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 16:31:44.386882 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":156,"threads_started":1}
I20260925 16:31:44.388774 21259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:44.389854 21259 log.cc:824] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:44.392588 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490: No bootstrap required, opened a new log
I20260925 16:31:44.393146 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:44.398169 21259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824bbe4f87d741f38a6d01cf10ede490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46253 } }
I20260925 16:31:44.398576 21259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:44.398627 21259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824bbe4f87d741f38a6d01cf10ede490, State: Initialized, Role: FOLLOWER
I20260925 16:31:44.398641 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:44.396383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44947 (local address 127.0.0.1:46253)
0925 16:31:44.396667 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:44.396673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:44.396821 (+ 148us) server_negotiation.cc:407] Connection header received
0925 16:31:44.396972 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:44.396976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:44.397028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:44.397160 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:44.398116 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:44.398120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:44.398123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:44.398136 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:44.398172 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:44.398186 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:44.398300 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:44.398444 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 16:31:44.398496 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260925 16:31:44.399181 21259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824bbe4f87d741f38a6d01cf10ede490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46253 } }
I20260925 16:31:44.399343 21259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:44.399418 21259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:44.399535 21259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:44.400795 21259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824bbe4f87d741f38a6d01cf10ede490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46253 } }
I20260925 16:31:44.401190 21259 leader_election.cc:302] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824bbe4f87d741f38a6d01cf10ede490; no voters:
I20260925 16:31:44.401536 21259 leader_election.cc:288] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:44.401712 21269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:44.401974 21269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [term 1 LEADER]: Becoming Leader. State: Replica: 824bbe4f87d741f38a6d01cf10ede490, State: Running, Role: LEADER
I20260925 16:31:44.402345 21269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824bbe4f87d741f38a6d01cf10ede490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46253 } }
I20260925 16:31:44.403137 21259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:44.404386 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824bbe4f87d741f38a6d01cf10ede490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824bbe4f87d741f38a6d01cf10ede490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46253 } } }
I20260925 16:31:44.404553 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:44.404860 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824bbe4f87d741f38a6d01cf10ede490. Latest consensus state: current_term: 1 leader_uuid: "824bbe4f87d741f38a6d01cf10ede490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824bbe4f87d741f38a6d01cf10ede490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46253 } } }
I20260925 16:31:44.404964 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:44.405117 21278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:44.409240 21278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:44.417027 21278 catalog_manager.cc:1347] Generated new cluster ID: b763f06d30244bdf89d6ebb8c17161e8
I20260925 16:31:44.417145 21278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:44.429382 21278 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:44.429889 21278 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:44.436410 21278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824bbe4f87d741f38a6d01cf10ede490: Generated new TSK 0
I20260925 16:31:44.437487 21278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:44.452873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21191
2026-09-25T16:31:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:44 dist-test-slave-31kt krb5kdc[21290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:44 dist-test-slave-31kt krb5kdc[21290](info): set up 2 sockets
Sep 25 16:31:44 dist-test-slave-31kt krb5kdc[21290](info): commencing operation
krb5kdc: starting...
W20260925 16:31:45.430351 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:45 dist-test-slave-31kt krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:45.570225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:45.705327 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:45.705674 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:45.705720 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:45.710125 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:45.710217 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:45.710237 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:45.710253 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:45.710268 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:45.716230 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:45.717469 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:45.718657 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:31:45.729017 21303 server_base.cc:1034] running on GCE node
W20260925 16:31:45.728963 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:45.728955 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:45.728955 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:45.730046 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:45.731310 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:45.732545 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790353905732527 us; error 60 us; skew 500 ppm
Sep 25 16:31:45 dist-test-slave-31kt krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:45.736277 21303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:45.737591 21303 webserver.cc:460] Webserver started at http://127.0.0.1:36201/ using document root <none> and password file <none>
I20260925 16:31:45.738298 21303 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:45.738376 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:45.738595 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:45.741026 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "f0696e4b91ef49bfbd271326ce402a72"
format_stamp: "Formatted at 2026-09-25 16:31:45 on dist-test-slave-31kt"
server_key: "e2c2ca1daf8b0b2e8c4b22515b430b36"
server_key_iv: "5ecc63944975c5509250f29a1b18b846"
server_key_version: "encryptionkey@0"
I20260925 16:31:45.741660 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0696e4b91ef49bfbd271326ce402a72"
format_stamp: "Formatted at 2026-09-25 16:31:45 on dist-test-slave-31kt"
server_key: "e2c2ca1daf8b0b2e8c4b22515b430b36"
server_key_iv: "5ecc63944975c5509250f29a1b18b846"
server_key_version: "encryptionkey@0"
I20260925 16:31:45.747077 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:31:45.750712 21318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:45.752480 21303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 16:31:45.752673 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "f0696e4b91ef49bfbd271326ce402a72"
format_stamp: "Formatted at 2026-09-25 16:31:45 on dist-test-slave-31kt"
server_key: "e2c2ca1daf8b0b2e8c4b22515b430b36"
server_key_iv: "5ecc63944975c5509250f29a1b18b846"
server_key_version: "encryptionkey@0"
I20260925 16:31:45.752844 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:45.774853 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:45.778587 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:45.778810 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:45.788127 21303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40829
I20260925 16:31:45.788169 21370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40829 every 8 connection(s)
I20260925 16:31:45.789803 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:45.794162 21371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:45.799659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21303
I20260925 16:31:45.799868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:45.800222 392 external_mini_cluster.cc:1428] Setting key c8e8e03785a12104a661087b7169211c
I20260925 16:31:45.803679 21371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72: Bootstrap starting.
I20260925 16:31:45.806902 21371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:45.807916 21371 log.cc:824] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:45.810984 21371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72: No bootstrap required, opened a new log
Sep 25 16:31:45 dist-test-slave-31kt krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:31:45.815214 21371 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0696e4b91ef49bfbd271326ce402a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40829 } }
I20260925 16:31:45.815558 21371 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:45.815614 21371 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0696e4b91ef49bfbd271326ce402a72, State: Initialized, Role: FOLLOWER
I20260925 16:31:45.816226 21371 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0696e4b91ef49bfbd271326ce402a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40829 } }
I20260925 16:31:45.816396 21371 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:45.816468 21371 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:45.816589 21371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:45.817956 21371 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0696e4b91ef49bfbd271326ce402a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40829 } }
I20260925 16:31:45.818465 21371 leader_election.cc:302] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0696e4b91ef49bfbd271326ce402a72; no voters:
I20260925 16:31:45.818917 21371 leader_election.cc:288] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:45.819064 21376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:45.819306 21376 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [term 1 LEADER]: Becoming Leader. State: Replica: f0696e4b91ef49bfbd271326ce402a72, State: Running, Role: LEADER
I20260925 16:31:45.819789 21376 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0696e4b91ef49bfbd271326ce402a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40829 } }
I20260925 16:31:45.820386 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:45.802351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:39299 (local address 127.0.0.1:40829)
0925 16:31:45.803088 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:45.803110 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:45.803189 (+ 79us) server_negotiation.cc:407] Connection header received
0925 16:31:45.804134 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:45.804262 (+ 128us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:45.804966 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:45.805425 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:45.806429 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:45.807599 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:45.809047 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:45.809684 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:45.809947 (+ 263us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:45.812925 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:45.812932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:45.812961 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:45.813002 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:45.815831 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:45.816592 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:45.816600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:45.816608 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:45.816800 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:45.817158 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:45.817205 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:45.817206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:45.817838 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:45.818111 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:45.818497 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 16:31:45.818852 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":222,"threads_started":1}
I20260925 16:31:45.820633 21371 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:45.822077 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0696e4b91ef49bfbd271326ce402a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0696e4b91ef49bfbd271326ce402a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40829 } } }
I20260925 16:31:45.822306 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:45.823179 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0696e4b91ef49bfbd271326ce402a72. Latest consensus state: current_term: 1 leader_uuid: "f0696e4b91ef49bfbd271326ce402a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0696e4b91ef49bfbd271326ce402a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40829 } } }
I20260925 16:31:45.823386 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:45.823431 21387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:45.825155 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:45.827659 21387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:45.837634 21387 catalog_manager.cc:1347] Generated new cluster ID: b1d07a78c30d4ded802263f6861d8a25
I20260925 16:31:45.837869 21387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:45.839859 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:45.827466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:51867 (local address 127.0.0.1:40829)
0925 16:31:45.827609 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:45.827614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:45.827888 (+ 274us) server_negotiation.cc:407] Connection header received
0925 16:31:45.827948 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:45.827951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:45.828002 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:45.828083 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:45.828804 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:45.829398 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:45.830430 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:45.830760 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:45.830872 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:45.834109 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:45.834115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:45.834118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:45.834169 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:45.836716 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:45.837749 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:45.837795 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:45.837798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:45.837941 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:45.838485 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:45.838490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:45.838492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:45.838813 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:45.839048 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:45.839541 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 16:31:45.839682 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:31:45.853437 21387 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:45.855397 21387 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:45.868225 21387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0696e4b91ef49bfbd271326ce402a72: Generated new TSK 0
I20260925 16:31:45.869242 21387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:45.883181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21303
W20260925 16:31:45.890137 21288 connection.cc:535] client connection to 127.0.0.1:40829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40829): Connection reset by peer (error 104)
2026-09-25T16:31:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:45 dist-test-slave-31kt krb5kdc[21402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:45 dist-test-slave-31kt krb5kdc[21402](info): set up 2 sockets
Sep 25 16:31:45 dist-test-slave-31kt krb5kdc[21402](info): commencing operation
krb5kdc: starting...
W20260925 16:31:46.874357 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:46 dist-test-slave-31kt krb5kdc[21402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:47.013880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:47.144941 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:47.145292 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:47.145341 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:47.150154 21415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:47.150254 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:47.150273 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:47.150297 21415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:47.150312 21415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:47.156574 21415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:47.158308 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:47.159998 21415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:47.171654 21420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:47.171657 21421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:47.171747 21415 server_base.cc:1034] running on GCE node
W20260925 16:31:47.171705 21423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:47.173051 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:47.174731 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:47.176070 21415 hybrid_clock.cc:648] HybridClock initialized: now 1790353907176040 us; error 66 us; skew 500 ppm
Sep 25 16:31:47 dist-test-slave-31kt krb5kdc[21402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:47.179383 21415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:47.180467 21415 webserver.cc:460] Webserver started at http://127.0.0.1:40321/ using document root <none> and password file <none>
I20260925 16:31:47.181097 21415 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:47.181169 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:47.181373 21415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:47.183733 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "b85f04344dac4b678f4593824dca1b2b"
format_stamp: "Formatted at 2026-09-25 16:31:47 on dist-test-slave-31kt"
server_key: "fcd07fd71b8aa103a176e76f32c6a3d3"
server_key_iv: "49776d4f5894acaeed232e233c44a2ff"
server_key_version: "encryptionkey@0"
I20260925 16:31:47.184367 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "b85f04344dac4b678f4593824dca1b2b"
format_stamp: "Formatted at 2026-09-25 16:31:47 on dist-test-slave-31kt"
server_key: "fcd07fd71b8aa103a176e76f32c6a3d3"
server_key_iv: "49776d4f5894acaeed232e233c44a2ff"
server_key_version: "encryptionkey@0"
I20260925 16:31:47.188997 21415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260925 16:31:47.192292 21430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:47.194370 21415 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 16:31:47.194545 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "b85f04344dac4b678f4593824dca1b2b"
format_stamp: "Formatted at 2026-09-25 16:31:47 on dist-test-slave-31kt"
server_key: "fcd07fd71b8aa103a176e76f32c6a3d3"
server_key_iv: "49776d4f5894acaeed232e233c44a2ff"
server_key_version: "encryptionkey@0"
I20260925 16:31:47.194684 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:47.221433 21415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:47.225978 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:47.226220 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:47.235510 21415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42253
I20260925 16:31:47.235555 21482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42253 every 8 connection(s)
I20260925 16:31:47.236867 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:47.240540 21483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:47.243216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21415
I20260925 16:31:47.243420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:47.243731 392 external_mini_cluster.cc:1428] Setting key d6fa55fd31a08b298b5ccd4518ec89f9
I20260925 16:31:47.247763 21483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b: Bootstrap starting.
I20260925 16:31:47.250782 21483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:47.251776 21483 log.cc:824] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:47.254747 21483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b: No bootstrap required, opened a new log
I20260925 16:31:47.255681 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:47.245647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:47711 (local address 127.0.0.1:42253)
0925 16:31:47.246312 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:47.246334 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:47.246385 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:31:47.247278 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:47.247303 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:47.247829 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:47.248274 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:47.248758 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:47.250001 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:47.251343 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:47.251999 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:47.252159 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:47.253252 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:47.253259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:47.253280 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:47.253875 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:47.253996 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:47.254028 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:47.254152 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:47.254383 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 16:31:47.254569 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":176,"threads_started":1}
I20260925 16:31:47.259141 21483 raft_consensus.cc:348] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f04344dac4b678f4593824dca1b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260925 16:31:47.259444 21483 raft_consensus.cc:374] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:47.259500 21483 raft_consensus.cc:729] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85f04344dac4b678f4593824dca1b2b, State: Initialized, Role: FOLLOWER
I20260925 16:31:47.259725 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:47.260089 21483 consensus_queue.cc:260] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f04344dac4b678f4593824dca1b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260925 16:31:47.260272 21483 raft_consensus.cc:388] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:47.260349 21483 raft_consensus.cc:482] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:47.260460 21483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:47.261917 21483 raft_consensus.cc:504] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f04344dac4b678f4593824dca1b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260925 16:31:47.262355 21483 leader_election.cc:302] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b85f04344dac4b678f4593824dca1b2b; no voters:
I20260925 16:31:47.262790 21483 leader_election.cc:288] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:47.262967 21493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:47.263231 21493 raft_consensus.cc:686] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [term 1 LEADER]: Becoming Leader. State: Replica: b85f04344dac4b678f4593824dca1b2b, State: Running, Role: LEADER
I20260925 16:31:47.263835 21493 consensus_queue.cc:237] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f04344dac4b678f4593824dca1b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } }
I20260925 16:31:47.264304 21483 sys_catalog.cc:564] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:47.266206 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b85f04344dac4b678f4593824dca1b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f04344dac4b678f4593824dca1b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } } }
I20260925 16:31:47.266422 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:47.266695 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b85f04344dac4b678f4593824dca1b2b. Latest consensus state: current_term: 1 leader_uuid: "b85f04344dac4b678f4593824dca1b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85f04344dac4b678f4593824dca1b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42253 } } }
I20260925 16:31:47.267207 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:47.268932 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:47.262528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:40107 (local address 127.0.0.1:42253)
0925 16:31:47.262704 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:47.262710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:47.262937 (+ 227us) server_negotiation.cc:407] Connection header received
0925 16:31:47.263205 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:47.263209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:47.263291 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:47.263420 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:47.263950 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:47.264748 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:47.266165 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:47.266536 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:47.266656 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:47.268208 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:47.268213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:47.268216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:47.268235 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:47.268279 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:47.268295 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:47.268398 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:47.268707 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 16:31:47.268772 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 16:31:47.270247 21508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:47.270355 21508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:47.270468 21507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:47.271284 21507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:47.277397 21507 catalog_manager.cc:1347] Generated new cluster ID: 25aa63c338fd45e79ae1e26e3a65a3c0
I20260925 16:31:47.277501 21507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:47.299790 21507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:47.301146 21507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:47.310775 21507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b85f04344dac4b678f4593824dca1b2b: Generated new TSK 0
I20260925 16:31:47.312070 21507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:47.323518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21415
W20260925 16:31:47.330081 21400 connection.cc:535] client connection to 127.0.0.1:42253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42253): Connection reset by peer (error 104)
2026-09-25T16:31:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:47 dist-test-slave-31kt krb5kdc[21514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:47 dist-test-slave-31kt krb5kdc[21514](info): set up 2 sockets
Sep 25 16:31:47 dist-test-slave-31kt krb5kdc[21514](info): commencing operation
krb5kdc: starting...
W20260925 16:31:48.298377 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:48 dist-test-slave-31kt krb5kdc[21514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:48.442418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:48.588337 21527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:48.588932 21527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:48.589012 21527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:48.594549 21527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:48.594697 21527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:48.594729 21527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:48.594753 21527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:48.594774 21527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:48.602959 21527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:48.604620 21527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:48.606160 21527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:48.617632 21535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:48.617537 21532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:48.617828 21533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:48.617985 21527 server_base.cc:1034] running on GCE node
I20260925 16:31:48.618842 21527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:48.620566 21527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:48.621906 21527 hybrid_clock.cc:648] HybridClock initialized: now 1790353908621882 us; error 55 us; skew 500 ppm
Sep 25 16:31:48 dist-test-slave-31kt krb5kdc[21514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:48.636648 21527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:48.638373 21527 webserver.cc:460] Webserver started at http://127.0.0.1:33639/ using document root <none> and password file <none>
I20260925 16:31:48.639286 21527 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:48.639600 21527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:48.640034 21527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:48.643853 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "63cc67b967494e559225e85ba396bca9"
format_stamp: "Formatted at 2026-09-25 16:31:48 on dist-test-slave-31kt"
server_key: "916543ee90b8c819e928f7a27a6dabfb"
server_key_iv: "542acd083b31f06b9431548b37702805"
server_key_version: "encryptionkey@0"
I20260925 16:31:48.644809 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "63cc67b967494e559225e85ba396bca9"
format_stamp: "Formatted at 2026-09-25 16:31:48 on dist-test-slave-31kt"
server_key: "916543ee90b8c819e928f7a27a6dabfb"
server_key_iv: "542acd083b31f06b9431548b37702805"
server_key_version: "encryptionkey@0"
I20260925 16:31:48.650575 21527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:31:48.654666 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:48.656868 21527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:48.657145 21527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "63cc67b967494e559225e85ba396bca9"
format_stamp: "Formatted at 2026-09-25 16:31:48 on dist-test-slave-31kt"
server_key: "916543ee90b8c819e928f7a27a6dabfb"
server_key_iv: "542acd083b31f06b9431548b37702805"
server_key_version: "encryptionkey@0"
I20260925 16:31:48.657333 21527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:48.663954 21527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:48.669183 21527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:48.669482 21527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:48.679181 21527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34803
I20260925 16:31:48.679275 21594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34803 every 8 connection(s)
I20260925 16:31:48.680428 21527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:48.682457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21527
I20260925 16:31:48.682631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:48.682976 392 external_mini_cluster.cc:1428] Setting key bb4f69c4ba92e233c302dd88504781d1
I20260925 16:31:48.684393 21595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:48.690196 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:48.684825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:54299 (local address 127.0.0.1:34803)
0925 16:31:48.685438 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:48.685461 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:48.685512 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:31:48.686454 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:48.686479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:48.686900 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:48.687346 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:48.688443 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:48.688450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:48.688468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:48.688482 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:48.688557 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:48.688582 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:48.688760 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:48.688846 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 16:31:48.689075 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":190,"threads_started":1}
I20260925 16:31:48.692358 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9: Bootstrap starting.
I20260925 16:31:48.694466 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:48.695336 21595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:48.696184 21595 log.cc:824] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:48.698995 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9: No bootstrap required, opened a new log
I20260925 16:31:48.699800 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:48.697482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:53993 (local address 127.0.0.1:34803)
0925 16:31:48.697677 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:48.697683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:48.697991 (+ 308us) server_negotiation.cc:407] Connection header received
0925 16:31:48.698203 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:48.698206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:48.698273 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:48.698355 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:48.699315 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:48.699319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:48.699321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:48.699336 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:48.699375 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:48.699388 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:48.699495 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:48.699591 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 16:31:48.699641 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:31:48.703413 21595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cc67b967494e559225e85ba396bca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260925 16:31:48.703722 21595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:48.703787 21595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cc67b967494e559225e85ba396bca9, State: Initialized, Role: FOLLOWER
I20260925 16:31:48.704264 21595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cc67b967494e559225e85ba396bca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260925 16:31:48.704412 21595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:48.704483 21595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:48.704586 21595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:48.705649 21595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cc67b967494e559225e85ba396bca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260925 16:31:48.706074 21595 leader_election.cc:302] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63cc67b967494e559225e85ba396bca9; no voters:
I20260925 16:31:48.706497 21595 leader_election.cc:288] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:48.706596 21605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:48.706785 21605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [term 1 LEADER]: Becoming Leader. State: Replica: 63cc67b967494e559225e85ba396bca9, State: Running, Role: LEADER
I20260925 16:31:48.707268 21605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cc67b967494e559225e85ba396bca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260925 16:31:48.708050 21595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:48.709476 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63cc67b967494e559225e85ba396bca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cc67b967494e559225e85ba396bca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } } }
I20260925 16:31:48.709493 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63cc67b967494e559225e85ba396bca9. Latest consensus state: current_term: 1 leader_uuid: "63cc67b967494e559225e85ba396bca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cc67b967494e559225e85ba396bca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } } }
I20260925 16:31:48.709808 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:48.709867 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9 [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:48.713685 21620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:48.713821 21620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:48.713899 21618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:48.714687 21618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:48.723174 21618 catalog_manager.cc:1347] Generated new cluster ID: 36bdd27389bc4915a8664f5de1ae0ffb
I20260925 16:31:48.723280 21618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:48.736564 21618 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:48.737154 21618 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:48.750962 21618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63cc67b967494e559225e85ba396bca9: Generated new TSK 0
I20260925 16:31:48.751869 21618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:48.758836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21527
2026-09-25T16:31:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:48 dist-test-slave-31kt krb5kdc[21626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:48 dist-test-slave-31kt krb5kdc[21626](info): set up 2 sockets
Sep 25 16:31:48 dist-test-slave-31kt krb5kdc[21626](info): commencing operation
krb5kdc: starting...
W20260925 16:31:49.738229 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:49 dist-test-slave-31kt krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:49.882184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:50.018498 21639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:50.018844 21639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:50.018893 21639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:50.023550 21639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:50.023643 21639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:50.023662 21639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:50.023679 21639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:50.023694 21639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:50.030819 21639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:50.032505 21639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:50.033949 21639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:31:50.046303 21639 server_base.cc:1034] running on GCE node
W20260925 16:31:50.046361 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:50.046234 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:50.046234 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:50.047559 21639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:50.049311 21639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:50.050588 21639 hybrid_clock.cc:648] HybridClock initialized: now 1790353910050520 us; error 57 us; skew 500 ppm
Sep 25 16:31:50 dist-test-slave-31kt krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:50.054423 21639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:50.055862 21639 webserver.cc:460] Webserver started at http://127.0.0.1:38983/ using document root <none> and password file <none>
I20260925 16:31:50.056654 21639 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:50.056754 21639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:50.057037 21639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:50.060019 21639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "5a8a56172db940649becb4e7b724ca7f"
format_stamp: "Formatted at 2026-09-25 16:31:50 on dist-test-slave-31kt"
server_key: "30ac22c8c3295db54b28a3da6fa86eab"
server_key_iv: "cac3664bf0bba92c169a05cbf8379212"
server_key_version: "encryptionkey@0"
I20260925 16:31:50.060827 21639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a8a56172db940649becb4e7b724ca7f"
format_stamp: "Formatted at 2026-09-25 16:31:50 on dist-test-slave-31kt"
server_key: "30ac22c8c3295db54b28a3da6fa86eab"
server_key_iv: "cac3664bf0bba92c169a05cbf8379212"
server_key_version: "encryptionkey@0"
I20260925 16:31:50.066563 21639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 16:31:50.070636 21654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:50.072767 21639 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:50.073017 21639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "5a8a56172db940649becb4e7b724ca7f"
format_stamp: "Formatted at 2026-09-25 16:31:50 on dist-test-slave-31kt"
server_key: "30ac22c8c3295db54b28a3da6fa86eab"
server_key_iv: "cac3664bf0bba92c169a05cbf8379212"
server_key_version: "encryptionkey@0"
I20260925 16:31:50.073299 21639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:50.098070 21639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:50.102712 21639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:50.102972 21639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:50.112011 21639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39837
I20260925 16:31:50.112043 21706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39837 every 8 connection(s)
I20260925 16:31:50.113421 21639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:50.116927 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:50.122453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21639
I20260925 16:31:50.122617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:50.122954 392 external_mini_cluster.cc:1428] Setting key 1a8608e2e903779f610289f045824481
I20260925 16:31:50.123554 21707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f: Bootstrap starting.
I20260925 16:31:50.126598 21707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:50.127900 21707 log.cc:824] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:50.131615 21707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f: No bootstrap required, opened a new log
Sep 25 16:31:50 dist-test-slave-31kt krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 16:31:50.136626 21707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8a56172db940649becb4e7b724ca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39837 } }
I20260925 16:31:50.136958 21707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:50.137027 21707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8a56172db940649becb4e7b724ca7f, State: Initialized, Role: FOLLOWER
I20260925 16:31:50.137584 21707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8a56172db940649becb4e7b724ca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39837 } }
I20260925 16:31:50.137789 21707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:50.137868 21707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:50.137987 21707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:50.139096 21707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8a56172db940649becb4e7b724ca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39837 } }
I20260925 16:31:50.139532 21707 leader_election.cc:302] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a8a56172db940649becb4e7b724ca7f; no voters:
I20260925 16:31:50.139981 21707 leader_election.cc:288] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:50.140249 21712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:50.140563 21712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [term 1 LEADER]: Becoming Leader. State: Replica: 5a8a56172db940649becb4e7b724ca7f, State: Running, Role: LEADER
I20260925 16:31:50.141045 21712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8a56172db940649becb4e7b724ca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39837 } }
I20260925 16:31:50.141547 21707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:50.143586 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a8a56172db940649becb4e7b724ca7f. Latest consensus state: current_term: 1 leader_uuid: "5a8a56172db940649becb4e7b724ca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8a56172db940649becb4e7b724ca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39837 } } }
I20260925 16:31:50.143579 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a8a56172db940649becb4e7b724ca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8a56172db940649becb4e7b724ca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39837 } } }
I20260925 16:31:50.143780 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:50.143918 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:50.124924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:43025 (local address 127.0.0.1:39837)
0925 16:31:50.125547 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:50.125568 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:50.125631 (+ 63us) server_negotiation.cc:407] Connection header received
0925 16:31:50.126654 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:50.126680 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:50.127254 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:50.127703 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:50.128926 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:50.130462 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:50.131915 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:50.132652 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:50.132845 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:50.136553 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:50.136562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:50.136614 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:50.136670 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:50.139025 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:50.139888 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:50.139896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:50.139903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:50.140115 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:50.140894 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:50.140898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:50.140900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:50.141602 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:50.141893 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:50.142336 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 16:31:50.142632 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":207,"threads_started":1}
I20260925 16:31:50.143790 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:50.147398 21726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:50.147521 21726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:50.147621 21727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:50.147938 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:50.148543 21727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:50.155946 21727 catalog_manager.cc:1347] Generated new cluster ID: 284b348f0583490d8041d49d21d53e28
I20260925 16:31:50.156152 21727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:50.160544 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:50.150387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:36479 (local address 127.0.0.1:39837)
0925 16:31:50.150671 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:50.150677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:50.150692 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:31:50.150838 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:50.150842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:50.150898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:50.150967 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:50.151723 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:50.152703 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:50.153635 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:50.153984 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:50.154096 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:50.156357 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:50.156361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:50.156364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:50.156394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:50.158323 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:50.159002 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:50.159006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:50.159008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:50.159100 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:50.159488 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:50.159492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:50.159494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:50.159765 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:50.159930 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:50.160233 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 16:31:50.160354 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260925 16:31:50.179998 21727 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:50.181252 21727 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:50.190208 21727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a8a56172db940649becb4e7b724ca7f: Generated new TSK 0
I20260925 16:31:50.191269 21727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:50.197525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21639
W20260925 16:31:50.204459 21624 connection.cc:535] client connection to 127.0.0.1:39837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39837): Connection reset by peer (error 104)
2026-09-25T16:31:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:50 dist-test-slave-31kt krb5kdc[21738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:50 dist-test-slave-31kt krb5kdc[21738](info): set up 2 sockets
Sep 25 16:31:50 dist-test-slave-31kt krb5kdc[21738](info): commencing operation
krb5kdc: starting...
W20260925 16:31:51.178275 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:51 dist-test-slave-31kt krb5kdc[21738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:51.321223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:51.459851 21751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:51.460291 21751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:51.460362 21751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:51.465373 21751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:51.465534 21751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:51.465567 21751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:51.465590 21751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:51.465611 21751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:51.472470 21751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:51.474138 21751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:51.475500 21751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:51.486330 21756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:51.486332 21759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:51.486330 21757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:51.486900 21751 server_base.cc:1034] running on GCE node
I20260925 16:31:51.487785 21751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:51.489387 21751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:51.490762 21751 hybrid_clock.cc:648] HybridClock initialized: now 1790353911490720 us; error 62 us; skew 500 ppm
Sep 25 16:31:51 dist-test-slave-31kt krb5kdc[21738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:51.495239 21751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:51.496789 21751 webserver.cc:460] Webserver started at http://127.0.0.1:35783/ using document root <none> and password file <none>
I20260925 16:31:51.497850 21751 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:51.497953 21751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:51.498230 21751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:51.501067 21751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "33fe3eb56ff348b8b1868831b2272de3"
format_stamp: "Formatted at 2026-09-25 16:31:51 on dist-test-slave-31kt"
server_key: "98cc16164055a5becc358aab09405c41"
server_key_iv: "8280fee2cc35e6501b41df75ad188738"
server_key_version: "encryptionkey@0"
I20260925 16:31:51.501987 21751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "33fe3eb56ff348b8b1868831b2272de3"
format_stamp: "Formatted at 2026-09-25 16:31:51 on dist-test-slave-31kt"
server_key: "98cc16164055a5becc358aab09405c41"
server_key_iv: "8280fee2cc35e6501b41df75ad188738"
server_key_version: "encryptionkey@0"
I20260925 16:31:51.507783 21751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 16:31:51.511981 21766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:51.514120 21751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:31:51.514335 21751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "33fe3eb56ff348b8b1868831b2272de3"
format_stamp: "Formatted at 2026-09-25 16:31:51 on dist-test-slave-31kt"
server_key: "98cc16164055a5becc358aab09405c41"
server_key_iv: "8280fee2cc35e6501b41df75ad188738"
server_key_version: "encryptionkey@0"
I20260925 16:31:51.514506 21751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:51.528805 21751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:51.533324 21751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:51.533596 21751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:51.542906 21751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40305
I20260925 16:31:51.542945 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40305 every 8 connection(s)
I20260925 16:31:51.544203 21751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:51.547819 21819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:51.550782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21751
I20260925 16:31:51.550947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:51.551244 392 external_mini_cluster.cc:1428] Setting key b2e63c3c6a7f8f94e61fa081236a766b
I20260925 16:31:51.555516 21819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3: Bootstrap starting.
I20260925 16:31:51.558624 21819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:51.559708 21819 log.cc:824] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:51.562628 21819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3: No bootstrap required, opened a new log
I20260925 16:31:51.564330 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:51.553064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:36297 (local address 127.0.0.1:40305)
0925 16:31:51.553743 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:51.553794 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:51.554231 (+ 437us) server_negotiation.cc:407] Connection header received
0925 16:31:51.555315 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:51.555343 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:51.555888 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:51.556288 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:51.556759 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:51.558157 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:51.559595 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:51.560365 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:51.560576 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:51.561697 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:51.561704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:51.561728 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:51.562374 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:51.562508 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:51.562544 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:51.562694 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:51.562983 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 16:31:51.563204 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":266,"threads_started":1}
I20260925 16:31:51.567047 21819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe3eb56ff348b8b1868831b2272de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260925 16:31:51.567384 21819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:51.567445 21819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fe3eb56ff348b8b1868831b2272de3, State: Initialized, Role: FOLLOWER
I20260925 16:31:51.567955 21819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe3eb56ff348b8b1868831b2272de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260925 16:31:51.568120 21819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:51.568193 21819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:51.568301 21819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:51.568832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:51.569517 21819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe3eb56ff348b8b1868831b2272de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260925 16:31:51.569960 21819 leader_election.cc:302] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33fe3eb56ff348b8b1868831b2272de3; no voters:
I20260925 16:31:51.570341 21819 leader_election.cc:288] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:51.570540 21824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:51.570787 21824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [term 1 LEADER]: Becoming Leader. State: Replica: 33fe3eb56ff348b8b1868831b2272de3, State: Running, Role: LEADER
I20260925 16:31:51.571223 21824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe3eb56ff348b8b1868831b2272de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260925 16:31:51.572126 21819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:51.573282 21829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33fe3eb56ff348b8b1868831b2272de3. Latest consensus state: current_term: 1 leader_uuid: "33fe3eb56ff348b8b1868831b2272de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe3eb56ff348b8b1868831b2272de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } } }
I20260925 16:31:51.573455 21829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:51.573839 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33fe3eb56ff348b8b1868831b2272de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fe3eb56ff348b8b1868831b2272de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } } }
I20260925 16:31:51.573947 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:51.574536 21838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:51.576937 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:51.571624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:56565 (local address 127.0.0.1:40305)
0925 16:31:51.571796 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:51.571803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:51.572079 (+ 276us) server_negotiation.cc:407] Connection header received
0925 16:31:51.572247 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:51.572250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:51.572316 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:51.572388 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:51.572722 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:51.573540 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:51.574768 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:51.575116 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:51.575184 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:51.576423 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:51.576427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:51.576436 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:51.576451 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:51.576480 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:51.576491 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:51.576574 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:51.576718 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 16:31:51.576760 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":65}
I20260925 16:31:51.579568 21838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:51.585815 21838 catalog_manager.cc:1347] Generated new cluster ID: b6e7e6eaee664966a8bc61d945185f76
I20260925 16:31:51.585916 21838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:51.621834 21838 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:51.623028 21838 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:51.631112 21838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33fe3eb56ff348b8b1868831b2272de3: Generated new TSK 0
I20260925 16:31:51.632001 21838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:51.644680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21751
W20260925 16:31:51.651036 21736 connection.cc:535] client connection to 127.0.0.1:40305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40305): Connection reset by peer (error 104)
2026-09-25T16:31:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:51 dist-test-slave-31kt krb5kdc[21850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:51 dist-test-slave-31kt krb5kdc[21850](info): set up 2 sockets
Sep 25 16:31:51 dist-test-slave-31kt krb5kdc[21850](info): commencing operation
krb5kdc: starting...
W20260925 16:31:52.618335 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:52 dist-test-slave-31kt krb5kdc[21850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 16:31:52.758215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:52.893887 21863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:52.894318 21863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:52.894392 21863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:52.898999 21863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:52.899108 21863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:52.899137 21863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:52.899158 21863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:52.899178 21863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:52.905705 21863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:52.907313 21863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:52.908689 21863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:52.919847 21871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:52.919847 21868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:52.920079 21863 server_base.cc:1034] running on GCE node
W20260925 16:31:52.919847 21869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:52.921073 21863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:52.922482 21863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:52.923702 21863 hybrid_clock.cc:648] HybridClock initialized: now 1790353912923667 us; error 58 us; skew 500 ppm
Sep 25 16:31:52 dist-test-slave-31kt krb5kdc[21850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:52.927179 21863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 16:31:52.928781 21863 webserver.cc:460] Webserver started at http://127.0.0.1:45075/ using document root <none> and password file <none>
I20260925 16:31:52.929621 21863 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:52.929715 21863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:52.930006 21863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:52.932964 21863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "0b679fb187c6497c9e7ed2acc86df3a2"
format_stamp: "Formatted at 2026-09-25 16:31:52 on dist-test-slave-31kt"
server_key: "aede9e22bfab77204922f80739693679"
server_key_iv: "e890d9168bcd5842219e58b05f587d29"
server_key_version: "encryptionkey@0"
I20260925 16:31:52.933792 21863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b679fb187c6497c9e7ed2acc86df3a2"
format_stamp: "Formatted at 2026-09-25 16:31:52 on dist-test-slave-31kt"
server_key: "aede9e22bfab77204922f80739693679"
server_key_iv: "e890d9168bcd5842219e58b05f587d29"
server_key_version: "encryptionkey@0"
I20260925 16:31:52.939018 21863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 16:31:52.942551 21878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:52.944573 21863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:52.944779 21863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "0b679fb187c6497c9e7ed2acc86df3a2"
format_stamp: "Formatted at 2026-09-25 16:31:52 on dist-test-slave-31kt"
server_key: "aede9e22bfab77204922f80739693679"
server_key_iv: "e890d9168bcd5842219e58b05f587d29"
server_key_version: "encryptionkey@0"
I20260925 16:31:52.944937 21863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:52.950302 21863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:52.954306 21863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:52.954525 21863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:52.962590 21863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34251
I20260925 16:31:52.962646 21930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34251 every 8 connection(s)
I20260925 16:31:52.963717 21863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:52.966841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21863
I20260925 16:31:52.966998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:52.967303 392 external_mini_cluster.cc:1428] Setting key 84f4b40895815d0a6308d22d13431c53
I20260925 16:31:52.968034 21931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:52.975219 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:52.969484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:58573 (local address 127.0.0.1:34251)
0925 16:31:52.970138 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:52.970164 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:52.970225 (+ 61us) server_negotiation.cc:407] Connection header received
0925 16:31:52.971188 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:52.971229 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:52.971737 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:52.972254 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:52.973158 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:52.973167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:52.973187 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:52.973202 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:52.973281 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:52.973313 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:52.973511 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:52.973808 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 16:31:52.974020 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":198,"threads_started":1}
I20260925 16:31:52.977891 21931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2: Bootstrap starting.
I20260925 16:31:52.980074 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:52.981171 21931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:52.982347 21931 log.cc:824] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:52.985265 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:52.982743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:58849 (local address 127.0.0.1:34251)
0925 16:31:52.982997 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:52.983004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:52.983237 (+ 233us) server_negotiation.cc:407] Connection header received
0925 16:31:52.983385 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:52.983389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:52.983433 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:52.983573 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:52.984613 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:52.984617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:52.984620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:52.984634 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:52.984671 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:52.984684 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:52.984784 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:52.984904 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 16:31:52.984956 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 16:31:52.985822 21931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2: No bootstrap required, opened a new log
I20260925 16:31:52.990906 21931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260925 16:31:52.991425 21931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:52.991518 21931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b679fb187c6497c9e7ed2acc86df3a2, State: Initialized, Role: FOLLOWER
I20260925 16:31:52.992179 21931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260925 16:31:52.992421 21931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:52.992506 21931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:52.992640 21931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:52.994315 21931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260925 16:31:52.994839 21931 leader_election.cc:302] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b679fb187c6497c9e7ed2acc86df3a2; no voters:
I20260925 16:31:52.995328 21931 leader_election.cc:288] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:52.995560 21941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:52.995982 21941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [term 1 LEADER]: Becoming Leader. State: Replica: 0b679fb187c6497c9e7ed2acc86df3a2, State: Running, Role: LEADER
I20260925 16:31:52.996420 21941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260925 16:31:52.997092 21931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:52.998358 21943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b679fb187c6497c9e7ed2acc86df3a2. Latest consensus state: current_term: 1 leader_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } } }
I20260925 16:31:52.998576 21943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:52.999399 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:52.999861 21942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b679fb187c6497c9e7ed2acc86df3a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } } }
I20260925 16:31:53.000022 21942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:53.003688 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:53.011776 21950 catalog_manager.cc:1347] Generated new cluster ID: 076ad463ac804344bffdbe5a168d65d0
I20260925 16:31:53.011881 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:53.034016 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:53.034427 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:53.055951 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b679fb187c6497c9e7ed2acc86df3a2: Generated new TSK 0
I20260925 16:31:53.057313 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:53.072489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21863
2026-09-25T16:31:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:53 dist-test-slave-31kt krb5kdc[21962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:53 dist-test-slave-31kt krb5kdc[21962](info): set up 2 sockets
Sep 25 16:31:53 dist-test-slave-31kt krb5kdc[21962](info): commencing operation
krb5kdc: starting...
W20260925 16:31:54.062284 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:54 dist-test-slave-31kt krb5kdc[21962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:54.206010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:54.346858 21975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:54.347316 21975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:54.347370 21975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:54.352965 21975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:54.353101 21975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:54.353123 21975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:54.353142 21975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:54.353158 21975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:54.360558 21975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:54.361974 21975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:54.363257 21975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:54.375010 21981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:54.375031 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:54.375010 21980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:54.378413 21975 server_base.cc:1034] running on GCE node
I20260925 16:31:54.379889 21975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:54.381940 21975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:54.385391 21975 hybrid_clock.cc:648] HybridClock initialized: now 1790353914385302 us; error 95 us; skew 500 ppm
Sep 25 16:31:54 dist-test-slave-31kt krb5kdc[21962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:54.395988 21975 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:54.397310 21975 webserver.cc:460] Webserver started at http://127.0.98.62:36895/ using document root <none> and password file <none>
I20260925 16:31:54.398061 21975 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:54.398142 21975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:54.398377 21975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:54.400733 21975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "d17812ac276d4403a6dbe0a69344a56a"
format_stamp: "Formatted at 2026-09-25 16:31:54 on dist-test-slave-31kt"
server_key: "95bee20141b6aca755797d01da114d8a"
server_key_iv: "bd97670f4f27bd26683db76ae4743f1d"
server_key_version: "encryptionkey@0"
I20260925 16:31:54.401384 21975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d17812ac276d4403a6dbe0a69344a56a"
format_stamp: "Formatted at 2026-09-25 16:31:54 on dist-test-slave-31kt"
server_key: "95bee20141b6aca755797d01da114d8a"
server_key_iv: "bd97670f4f27bd26683db76ae4743f1d"
server_key_version: "encryptionkey@0"
I20260925 16:31:54.406427 21975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:31:54.410013 21990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:54.411942 21975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:54.412148 21975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "d17812ac276d4403a6dbe0a69344a56a"
format_stamp: "Formatted at 2026-09-25 16:31:54 on dist-test-slave-31kt"
server_key: "95bee20141b6aca755797d01da114d8a"
server_key_iv: "bd97670f4f27bd26683db76ae4743f1d"
server_key_version: "encryptionkey@0"
I20260925 16:31:54.412304 21975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:54.437156 21975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:54.440858 21975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:54.441088 21975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:54.449440 21975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41187
I20260925 16:31:54.449558 22042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41187 every 8 connection(s)
I20260925 16:31:54.450644 21975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:54.454793 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:54.456326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 21975
I20260925 16:31:54.456473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:54.456766 392 external_mini_cluster.cc:1428] Setting key bf94c82b6b9c868d7f53572bf03b67a0
I20260925 16:31:54.463536 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a: Bootstrap starting.
I20260925 16:31:54.467437 22043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a: Neither blocks nor log segments found. Creating new log.
Sep 25 16:31:54 dist-test-slave-31kt krb5kdc[21962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:54.468752 22043 log.cc:824] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:54.472412 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a: No bootstrap required, opened a new log
I20260925 16:31:54.477089 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:54.458578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56655 (local address 127.0.98.62:41187)
0925 16:31:54.459304 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:54.459324 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:54.459378 (+ 54us) server_negotiation.cc:407] Connection header received
0925 16:31:54.460323 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:54.460352 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:54.460867 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:54.461331 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:54.462673 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:54.463901 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:54.464902 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:54.465677 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:54.465962 (+ 285us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:54.469873 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:54.469882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:54.469904 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:54.469948 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:54.472289 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:54.473179 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:54.473187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:54.473197 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:54.473407 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:54.473844 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:54.473848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:54.473849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:54.474573 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:54.474845 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:54.475322 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 16:31:54.475691 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":218,"threads_started":1}
I20260925 16:31:54.478245 22043 raft_consensus.cc:348] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17812ac276d4403a6dbe0a69344a56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260925 16:31:54.478633 22043 raft_consensus.cc:374] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:54.478689 22043 raft_consensus.cc:729] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17812ac276d4403a6dbe0a69344a56a, State: Initialized, Role: FOLLOWER
I20260925 16:31:54.479283 22043 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17812ac276d4403a6dbe0a69344a56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260925 16:31:54.479442 22043 raft_consensus.cc:388] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:54.479512 22043 raft_consensus.cc:482] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:54.479641 22043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:54.481344 22043 raft_consensus.cc:504] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17812ac276d4403a6dbe0a69344a56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260925 16:31:54.482080 22043 leader_election.cc:302] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17812ac276d4403a6dbe0a69344a56a; no voters:
I20260925 16:31:54.482558 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:54.482625 22043 leader_election.cc:288] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:54.482837 22048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:54.483263 22048 raft_consensus.cc:686] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [term 1 LEADER]: Becoming Leader. State: Replica: d17812ac276d4403a6dbe0a69344a56a, State: Running, Role: LEADER
I20260925 16:31:54.483834 22048 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17812ac276d4403a6dbe0a69344a56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260925 16:31:54.484421 22043 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:54.485729 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17812ac276d4403a6dbe0a69344a56a. Latest consensus state: current_term: 1 leader_uuid: "d17812ac276d4403a6dbe0a69344a56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17812ac276d4403a6dbe0a69344a56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } } }
I20260925 16:31:54.485960 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:54.486244 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17812ac276d4403a6dbe0a69344a56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17812ac276d4403a6dbe0a69344a56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } } }
I20260925 16:31:54.486364 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:54.491223 22068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:54.491349 22068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:54.491441 22061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:54.492524 22061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:54.499732 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:54.486306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38843 (local address 127.0.98.62:41187)
0925 16:31:54.486441 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:54.486446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:54.486986 (+ 540us) server_negotiation.cc:407] Connection header received
0925 16:31:54.487624 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:54.487628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:54.487693 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:54.487746 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:54.488866 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:54.489706 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:54.491186 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:54.491660 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:54.491829 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:54.494033 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:54.494038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:54.494042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:54.494100 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:54.496327 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:54.497297 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:54.497303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:54.497305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:54.497514 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:54.498403 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:54.498409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:54.498411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:54.498744 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:54.498949 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:54.499363 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 16:31:54.499519 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:31:54.500356 22061 catalog_manager.cc:1347] Generated new cluster ID: 0bbe68ef7f22445cb0f0d17120ededb4
I20260925 16:31:54.500445 22061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:54.508121 22061 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:54.509408 22061 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:54.517354 22061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d17812ac276d4403a6dbe0a69344a56a: Generated new TSK 0
I20260925 16:31:54.518508 22061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:54.531704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 21975
W20260925 16:31:54.537896 21960 connection.cc:535] client connection to 127.0.98.62:41187 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41187): Connection reset by peer (error 104)
2026-09-25T16:31:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:54 dist-test-slave-31kt krb5kdc[22074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:54 dist-test-slave-31kt krb5kdc[22074](info): set up 2 sockets
Sep 25 16:31:54 dist-test-slave-31kt krb5kdc[22074](info): commencing operation
krb5kdc: starting...
W20260925 16:31:55.506193 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:55 dist-test-slave-31kt krb5kdc[22074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:55.641103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:55.779623 22087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:55.780037 22087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:55.780108 22087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:55.784862 22087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:55.784977 22087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:55.785004 22087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:55.785028 22087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:55.785049 22087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:55.791486 22087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:55.793012 22087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:55.794365 22087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:31:55.804737 22087 server_base.cc:1034] running on GCE node
W20260925 16:31:55.804699 22093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:55.804710 22092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:55.804678 22095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:55.805948 22087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:55.807466 22087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:55.808689 22087 hybrid_clock.cc:648] HybridClock initialized: now 1790353915808666 us; error 52 us; skew 500 ppm
Sep 25 16:31:55 dist-test-slave-31kt krb5kdc[22074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:55.812065 22087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:55.813318 22087 webserver.cc:460] Webserver started at http://127.0.98.62:41125/ using document root <none> and password file <none>
I20260925 16:31:55.814090 22087 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:55.814177 22087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:55.814404 22087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:55.817123 22087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "4d4d739278264cb2ac0ac9eef294db66"
format_stamp: "Formatted at 2026-09-25 16:31:55 on dist-test-slave-31kt"
server_key: "b0c886e3b0d591f3683b7c22ddc68295"
server_key_iv: "942971ab8a3de38fa6cd19df21ea4905"
server_key_version: "encryptionkey@0"
I20260925 16:31:55.817883 22087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d4d739278264cb2ac0ac9eef294db66"
format_stamp: "Formatted at 2026-09-25 16:31:55 on dist-test-slave-31kt"
server_key: "b0c886e3b0d591f3683b7c22ddc68295"
server_key_iv: "942971ab8a3de38fa6cd19df21ea4905"
server_key_version: "encryptionkey@0"
I20260925 16:31:55.823028 22087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 16:31:55.826748 22102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:55.828778 22087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:55.828984 22087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "4d4d739278264cb2ac0ac9eef294db66"
format_stamp: "Formatted at 2026-09-25 16:31:55 on dist-test-slave-31kt"
server_key: "b0c886e3b0d591f3683b7c22ddc68295"
server_key_iv: "942971ab8a3de38fa6cd19df21ea4905"
server_key_version: "encryptionkey@0"
I20260925 16:31:55.829188 22087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:55.846972 22087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:55.852171 22087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:55.852432 22087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:55.862356 22087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38197
I20260925 16:31:55.862388 22154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38197 every 8 connection(s)
I20260925 16:31:55.863783 22087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:55.867703 22155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:55.870352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22087
I20260925 16:31:55.870513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:55.870882 392 external_mini_cluster.cc:1428] Setting key 9ae2acc99affbbd942115608f7eca8bf
I20260925 16:31:55.875007 22155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66: Bootstrap starting.
I20260925 16:31:55.877673 22155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:55.878708 22155 log.cc:824] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:55.882215 22155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66: No bootstrap required, opened a new log
I20260925 16:31:55.884095 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:55.872561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57317 (local address 127.0.98.62:38197)
0925 16:31:55.873298 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:55.873322 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:55.873386 (+ 64us) server_negotiation.cc:407] Connection header received
0925 16:31:55.874434 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:55.874468 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:55.875047 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:55.875549 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:55.876142 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:55.877407 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:55.878754 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:55.879514 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:55.879764 (+ 250us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:55.881191 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:55.881199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:55.881223 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:55.881908 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:55.882003 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:55.882040 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:55.882202 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:55.882581 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 16:31:55.882816 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":183,"threads_started":1}
I20260925 16:31:55.886879 22155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4d739278264cb2ac0ac9eef294db66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38197 } }
I20260925 16:31:55.887210 22155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:55.887249 22155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4d739278264cb2ac0ac9eef294db66, State: Initialized, Role: FOLLOWER
I20260925 16:31:55.887800 22155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4d739278264cb2ac0ac9eef294db66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38197 } }
I20260925 16:31:55.887959 22155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:55.888032 22155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:55.888139 22155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:55.888645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:55.889287 22155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4d739278264cb2ac0ac9eef294db66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38197 } }
I20260925 16:31:55.889694 22155 leader_election.cc:302] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d4d739278264cb2ac0ac9eef294db66; no voters:
I20260925 16:31:55.890097 22155 leader_election.cc:288] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:55.890194 22160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:55.890470 22160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [term 1 LEADER]: Becoming Leader. State: Replica: 4d4d739278264cb2ac0ac9eef294db66, State: Running, Role: LEADER
I20260925 16:31:55.890889 22160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4d739278264cb2ac0ac9eef294db66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38197 } }
I20260925 16:31:55.891851 22155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:55.893278 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d4d739278264cb2ac0ac9eef294db66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4d739278264cb2ac0ac9eef294db66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38197 } } }
I20260925 16:31:55.893479 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:55.893832 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d4d739278264cb2ac0ac9eef294db66. Latest consensus state: current_term: 1 leader_uuid: "4d4d739278264cb2ac0ac9eef294db66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4d739278264cb2ac0ac9eef294db66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38197 } } }
I20260925 16:31:55.893918 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:55.894438 22174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:55.896950 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:55.891520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37831 (local address 127.0.98.62:38197)
0925 16:31:55.891664 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:55.891670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:55.891904 (+ 234us) server_negotiation.cc:407] Connection header received
0925 16:31:55.892078 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:55.892081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:55.892134 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:55.892209 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:55.892531 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:55.893377 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:55.894640 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:55.894960 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:55.895076 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:55.896167 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:55.896171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:55.896174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:55.896193 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:55.896230 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:55.896245 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:55.896331 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:55.896725 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 16:31:55.896775 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:31:55.898518 22174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:55.906860 22174 catalog_manager.cc:1347] Generated new cluster ID: 7fc478843a4842618b08ebfd3f5111df
I20260925 16:31:55.907019 22174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:55.936744 22174 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:55.938661 22174 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:55.952881 22174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d4d739278264cb2ac0ac9eef294db66: Generated new TSK 0
I20260925 16:31:55.954401 22174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:55.964620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22087
W20260925 16:31:55.970255 22072 connection.cc:535] client connection to 127.0.98.62:38197 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38197): Connection reset by peer (error 104)
2026-09-25T16:31:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:56 dist-test-slave-31kt krb5kdc[22186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:56 dist-test-slave-31kt krb5kdc[22186](info): set up 2 sockets
Sep 25 16:31:56 dist-test-slave-31kt krb5kdc[22186](info): commencing operation
krb5kdc: starting...
W20260925 16:31:56.938175 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:57 dist-test-slave-31kt krb5kdc[22186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:57.078606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:57.211952 22199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:57.212370 22199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:57.212421 22199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:57.217072 22199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:57.217190 22199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:57.217211 22199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:57.217228 22199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:57.217244 22199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:57.224017 22199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:57.225612 22199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:57.227041 22199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:31:57.238441 22199 server_base.cc:1034] running on GCE node
W20260925 16:31:57.238380 22204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:57.238380 22207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:57.238402 22205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:57.239550 22199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:57.241278 22199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:57.242571 22199 hybrid_clock.cc:648] HybridClock initialized: now 1790353917242534 us; error 66 us; skew 500 ppm
Sep 25 16:31:57 dist-test-slave-31kt krb5kdc[22186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:57.246637 22199 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:57.248201 22199 webserver.cc:460] Webserver started at http://127.0.98.62:39177/ using document root <none> and password file <none>
I20260925 16:31:57.249002 22199 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:57.249094 22199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:57.249380 22199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:57.252302 22199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "0df17979aa3f4b98990eab41b5fa8251"
format_stamp: "Formatted at 2026-09-25 16:31:57 on dist-test-slave-31kt"
server_key: "60a343a477601c6f72874d0a6785303e"
server_key_iv: "b5324789d6072c2fbc22f89443c52b55"
server_key_version: "encryptionkey@0"
I20260925 16:31:57.253067 22199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "0df17979aa3f4b98990eab41b5fa8251"
format_stamp: "Formatted at 2026-09-25 16:31:57 on dist-test-slave-31kt"
server_key: "60a343a477601c6f72874d0a6785303e"
server_key_iv: "b5324789d6072c2fbc22f89443c52b55"
server_key_version: "encryptionkey@0"
I20260925 16:31:57.258594 22199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:31:57.262389 22214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:57.264292 22199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:31:57.264513 22199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "0df17979aa3f4b98990eab41b5fa8251"
format_stamp: "Formatted at 2026-09-25 16:31:57 on dist-test-slave-31kt"
server_key: "60a343a477601c6f72874d0a6785303e"
server_key_iv: "b5324789d6072c2fbc22f89443c52b55"
server_key_version: "encryptionkey@0"
I20260925 16:31:57.264688 22199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:57.270728 22199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:57.274569 22199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:57.274794 22199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:57.284276 22199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43391
I20260925 16:31:57.284333 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43391 every 8 connection(s)
I20260925 16:31:57.285643 22199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:57.287763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22199
I20260925 16:31:57.288053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:57.288554 392 external_mini_cluster.cc:1428] Setting key 4a89698e5d4a364558ad67204daf1a14
I20260925 16:31:57.290402 22267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:57.296111 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:57.290576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34711 (local address 127.0.98.62:43391)
0925 16:31:57.291175 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:57.291196 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:57.291259 (+ 63us) server_negotiation.cc:407] Connection header received
0925 16:31:57.292113 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:57.292136 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:57.292580 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:57.293042 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:57.294288 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:57.294295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:57.294313 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:57.294327 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:57.294408 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:57.294434 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:57.294602 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:57.294736 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 16:31:57.294973 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":179,"threads_started":1}
I20260925 16:31:57.298970 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251: Bootstrap starting.
I20260925 16:31:57.300426 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:57.302608 22267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:57.303894 22267 log.cc:824] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251: Log is configured to *not* fsync() on all Append() calls
I20260925 16:31:57.305030 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:57.303125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36357 (local address 127.0.98.62:43391)
0925 16:31:57.303298 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:57.303304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:57.303464 (+ 160us) server_negotiation.cc:407] Connection header received
0925 16:31:57.303610 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:57.303613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:57.303667 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:57.303777 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:57.304606 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:57.304609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:57.304611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:31:57.304622 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:57.304651 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:31:57.304663 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:57.304756 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:57.304838 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 16:31:57.304881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 16:31:57.307435 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251: No bootstrap required, opened a new log
I20260925 16:31:57.313010 22267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17979aa3f4b98990eab41b5fa8251" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 16:31:57.313444 22267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:57.313488 22267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df17979aa3f4b98990eab41b5fa8251, State: Initialized, Role: FOLLOWER
I20260925 16:31:57.314174 22267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17979aa3f4b98990eab41b5fa8251" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 16:31:57.314371 22267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:57.314441 22267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:57.314543 22267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:57.316067 22267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17979aa3f4b98990eab41b5fa8251" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 16:31:57.316615 22267 leader_election.cc:302] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df17979aa3f4b98990eab41b5fa8251; no voters:
I20260925 16:31:57.317004 22267 leader_election.cc:288] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:57.317214 22277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:57.317597 22277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [term 1 LEADER]: Becoming Leader. State: Replica: 0df17979aa3f4b98990eab41b5fa8251, State: Running, Role: LEADER
I20260925 16:31:57.318104 22277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17979aa3f4b98990eab41b5fa8251" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 16:31:57.318532 22267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:57.320209 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df17979aa3f4b98990eab41b5fa8251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17979aa3f4b98990eab41b5fa8251" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } } }
I20260925 16:31:57.320410 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:57.320758 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df17979aa3f4b98990eab41b5fa8251. Latest consensus state: current_term: 1 leader_uuid: "0df17979aa3f4b98990eab41b5fa8251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df17979aa3f4b98990eab41b5fa8251" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } } }
I20260925 16:31:57.320845 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:57.320863 22290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:57.324640 22290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:57.331970 22290 catalog_manager.cc:1347] Generated new cluster ID: 838d9749a8ed4e9f8ae6d386b7b69c04
I20260925 16:31:57.332101 22290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:57.349398 22290 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:57.349886 22290 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:57.356791 22290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df17979aa3f4b98990eab41b5fa8251: Generated new TSK 0
I20260925 16:31:57.357925 22290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:57.372201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22199
2026-09-25T16:31:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:57 dist-test-slave-31kt krb5kdc[22298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:57 dist-test-slave-31kt krb5kdc[22298](info): set up 2 sockets
Sep 25 16:31:57 dist-test-slave-31kt krb5kdc[22298](info): commencing operation
krb5kdc: starting...
W20260925 16:31:58.358294 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:58 dist-test-slave-31kt krb5kdc[22298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:58.494988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:31:58.637914 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:31:58.638340 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:31:58.638407 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:31:58.643167 22311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:31:58.643262 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:31:58.643289 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:31:58.643312 22311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:31:58.643335 22311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:31:58.649746 22311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:31:58.651321 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:31:58.652745 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:31:58.663754 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:31:58.663759 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:58.663831 22311 server_base.cc:1034] running on GCE node
W20260925 16:31:58.663759 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:31:58.664942 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:31:58.666386 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:31:58.667587 22311 hybrid_clock.cc:648] HybridClock initialized: now 1790353918667566 us; error 53 us; skew 500 ppm
Sep 25 16:31:58 dist-test-slave-31kt krb5kdc[22298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:31:58.671434 22311 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:31:58.672603 22311 webserver.cc:460] Webserver started at http://127.0.98.62:37169/ using document root <none> and password file <none>
I20260925 16:31:58.673276 22311 fs_manager.cc:359] Metadata directory not provided
I20260925 16:31:58.673355 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:31:58.673581 22311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:31:58.675894 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4"
format_stamp: "Formatted at 2026-09-25 16:31:58 on dist-test-slave-31kt"
server_key: "4b91a9baf4fbd59e9488b7546d3bc4a5"
server_key_iv: "f3cd8e2bce7dbee76a173426b0917f7b"
server_key_version: "encryptionkey@0"
I20260925 16:31:58.676499 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4"
format_stamp: "Formatted at 2026-09-25 16:31:58 on dist-test-slave-31kt"
server_key: "4b91a9baf4fbd59e9488b7546d3bc4a5"
server_key_iv: "f3cd8e2bce7dbee76a173426b0917f7b"
server_key_version: "encryptionkey@0"
I20260925 16:31:58.681373 22311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 16:31:58.684880 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:31:58.686759 22311 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 16:31:58.686951 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4"
format_stamp: "Formatted at 2026-09-25 16:31:58 on dist-test-slave-31kt"
server_key: "4b91a9baf4fbd59e9488b7546d3bc4a5"
server_key_iv: "f3cd8e2bce7dbee76a173426b0917f7b"
server_key_version: "encryptionkey@0"
I20260925 16:31:58.687093 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:31:58.710876 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:31:58.714666 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:31:58.714932 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:31:58.725687 22311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43179
I20260925 16:31:58.725744 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43179 every 8 connection(s)
I20260925 16:31:58.726977 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:31:58.731431 22379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:31:58.734606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22311
I20260925 16:31:58.734768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:31:58.735095 392 external_mini_cluster.cc:1428] Setting key 61bb8390ded1ffb4bea29d7e4711ee8f
I20260925 16:31:58.741523 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4: Bootstrap starting.
I20260925 16:31:58.745127 22379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4: Neither blocks nor log segments found. Creating new log.
I20260925 16:31:58.746553 22379 log.cc:824] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4: Log is configured to *not* fsync() on all Append() calls
Sep 25 16:31:58 dist-test-slave-31kt krb5kdc[22298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:31:58.750407 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4: No bootstrap required, opened a new log
I20260925 16:31:58.755615 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:58.737014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49475 (local address 127.0.98.62:43179)
0925 16:31:58.737801 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:58.737827 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:58.738077 (+ 250us) server_negotiation.cc:407] Connection header received
0925 16:31:58.739111 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:58.739135 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:58.739570 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:58.739949 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:58.741091 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:58.742320 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:58.743286 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:58.743889 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:58.744060 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:58.747970 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:58.747981 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:58.748011 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:58.748054 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:58.750789 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:58.751723 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:58.751732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:58.751740 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:58.751951 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:58.752352 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:58.752355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:58.752357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:58.753097 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:58.753397 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:58.753729 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 16:31:58.754237 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":291,"threads_started":1}
I20260925 16:31:58.756196 22379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260925 16:31:58.756616 22379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:31:58.756675 22379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb01b1b6a264d8fb31ee93b9b07c6c4, State: Initialized, Role: FOLLOWER
I20260925 16:31:58.757339 22379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260925 16:31:58.757516 22379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:31:58.757594 22379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:31:58.757711 22379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:31:58.759339 22379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260925 16:31:58.759871 22379 leader_election.cc:302] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fb01b1b6a264d8fb31ee93b9b07c6c4; no voters:
I20260925 16:31:58.760154 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:31:58.760371 22379 leader_election.cc:288] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:31:58.760543 22384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:31:58.760767 22384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb01b1b6a264d8fb31ee93b9b07c6c4, State: Running, Role: LEADER
I20260925 16:31:58.761313 22384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260925 16:31:58.762329 22379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:31:58.763064 22384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fb01b1b6a264d8fb31ee93b9b07c6c4. Latest consensus state: current_term: 1 leader_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } } }
I20260925 16:31:58.763224 22384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [sys.catalog]: This master's current role is: LEADER
I20260925 16:31:58.763499 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb01b1b6a264d8fb31ee93b9b07c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } } }
I20260925 16:31:58.763794 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4 [sys.catalog]: This master's current role is: LEADER
W20260925 16:31:58.769595 22404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:31:58.769855 22404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:31:58.769996 22395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:31:58.770979 22395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:31:58.772895 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:31:58.762687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51057 (local address 127.0.98.62:43179)
0925 16:31:58.762837 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 16:31:58.762842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:31:58.763345 (+ 503us) server_negotiation.cc:407] Connection header received
0925 16:31:58.763576 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:31:58.763579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:31:58.763723 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:31:58.763798 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:31:58.764755 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:58.765527 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:58.766734 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:31:58.767036 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:31:58.767171 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:31:58.769105 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:31:58.769109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:31:58.769111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:31:58.769143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:31:58.770844 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:58.771438 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:58.771442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:58.771444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:58.771542 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:31:58.771851 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:31:58.771855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:31:58.771857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:31:58.772105 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:31:58.772245 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:31:58.772523 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 16:31:58.772677 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 16:31:58.778971 22395 catalog_manager.cc:1347] Generated new cluster ID: 640bd439870944f4bcc9f186e5617328
I20260925 16:31:58.779105 22395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:31:58.793465 22395 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:31:58.795323 22395 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:31:58.805496 22395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fb01b1b6a264d8fb31ee93b9b07c6c4: Generated new TSK 0
I20260925 16:31:58.806577 22395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:31:58.816736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22311
2026-09-25T16:31:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:31:58 dist-test-slave-31kt krb5kdc[22410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:31:58 dist-test-slave-31kt krb5kdc[22410](info): set up 2 sockets
Sep 25 16:31:58 dist-test-slave-31kt krb5kdc[22410](info): commencing operation
krb5kdc: starting...
W20260925 16:31:59.790683 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:31:59 dist-test-slave-31kt krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:31:59.925262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:00.062773 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:00.063205 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:00.063257 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:00.068418 22423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:00.068572 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:00.068594 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:00.068612 22423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:00.068629 22423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:00.075811 22423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:00.077519 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:00.079093 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:00.090510 22428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:00.090468 22429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:00.090553 22431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:00.091293 22423 server_base.cc:1034] running on GCE node
I20260925 16:32:00.092154 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:00.093912 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:00.095152 22423 hybrid_clock.cc:648] HybridClock initialized: now 1790353920095124 us; error 57 us; skew 500 ppm
Sep 25 16:32:00 dist-test-slave-31kt krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:00.105530 22423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:00.107622 22423 webserver.cc:460] Webserver started at http://127.0.98.62:42223/ using document root <none> and password file <none>
I20260925 16:32:00.108623 22423 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:00.108753 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:00.109005 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:00.111493 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "c66f88c91b944ae7981ae5be8e1e042b"
format_stamp: "Formatted at 2026-09-25 16:32:00 on dist-test-slave-31kt"
server_key: "87166707486424d3bfd91f381c7198fd"
server_key_iv: "73d4280a4ec163463bbad6be8df8ef19"
server_key_version: "encryptionkey@0"
I20260925 16:32:00.112159 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c66f88c91b944ae7981ae5be8e1e042b"
format_stamp: "Formatted at 2026-09-25 16:32:00 on dist-test-slave-31kt"
server_key: "87166707486424d3bfd91f381c7198fd"
server_key_iv: "73d4280a4ec163463bbad6be8df8ef19"
server_key_version: "encryptionkey@0"
I20260925 16:32:00.117384 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:32:00.121876 22438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:00.124101 22423 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:00.124316 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "c66f88c91b944ae7981ae5be8e1e042b"
format_stamp: "Formatted at 2026-09-25 16:32:00 on dist-test-slave-31kt"
server_key: "87166707486424d3bfd91f381c7198fd"
server_key_iv: "73d4280a4ec163463bbad6be8df8ef19"
server_key_version: "encryptionkey@0"
I20260925 16:32:00.124509 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:00.149278 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:00.154407 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:00.154654 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:00.163506 22423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40265
I20260925 16:32:00.163560 22490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40265 every 8 connection(s)
I20260925 16:32:00.164753 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:00.168779 22491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:00.175092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22423
I20260925 16:32:00.175254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:00.175559 392 external_mini_cluster.cc:1428] Setting key ad3c4d2d624e0ef995f33512365bb2d7
I20260925 16:32:00.176677 22491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b: Bootstrap starting.
I20260925 16:32:00.180532 22491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:00.181691 22491 log.cc:824] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:00.184983 22491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b: No bootstrap required, opened a new log
I20260925 16:32:00.189278 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:00.177570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34445 (local address 127.0.98.62:40265)
0925 16:32:00.178292 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:00.178318 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:00.178392 (+ 74us) server_negotiation.cc:407] Connection header received
0925 16:32:00.179563 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:00.179597 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:00.180188 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:00.180714 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:00.181293 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:00.182772 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:00.184116 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:00.184944 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:00.185157 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:00.186349 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:00.186357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:00.186382 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:32:00.187070 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:00.187169 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:32:00.187208 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:00.187338 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:00.187567 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 16:32:00.187881 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":248,"threads_started":1}
I20260925 16:32:00.190678 22491 raft_consensus.cc:348] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f88c91b944ae7981ae5be8e1e042b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260925 16:32:00.191028 22491 raft_consensus.cc:374] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:00.191067 22491 raft_consensus.cc:729] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f88c91b944ae7981ae5be8e1e042b, State: Initialized, Role: FOLLOWER
I20260925 16:32:00.191671 22491 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f88c91b944ae7981ae5be8e1e042b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260925 16:32:00.191843 22491 raft_consensus.cc:388] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:32:00.191915 22491 raft_consensus.cc:482] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:32:00.192040 22491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:00.193526 22491 raft_consensus.cc:504] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f88c91b944ae7981ae5be8e1e042b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260925 16:32:00.193841 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:32:00.194118 22491 leader_election.cc:302] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f88c91b944ae7981ae5be8e1e042b; no voters:
I20260925 16:32:00.194552 22491 leader_election.cc:288] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:32:00.194897 22496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:00.195391 22496 raft_consensus.cc:686] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [term 1 LEADER]: Becoming Leader. State: Replica: c66f88c91b944ae7981ae5be8e1e042b, State: Running, Role: LEADER
I20260925 16:32:00.196017 22496 consensus_queue.cc:237] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f88c91b944ae7981ae5be8e1e042b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260925 16:32:00.196457 22491 sys_catalog.cc:564] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:00.198386 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c66f88c91b944ae7981ae5be8e1e042b. Latest consensus state: current_term: 1 leader_uuid: "c66f88c91b944ae7981ae5be8e1e042b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f88c91b944ae7981ae5be8e1e042b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } } }
I20260925 16:32:00.198632 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:00.198357 22497 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c66f88c91b944ae7981ae5be8e1e042b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f88c91b944ae7981ae5be8e1e042b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } } }
I20260925 16:32:00.199013 22497 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:00.199129 22514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:00.203106 22514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:00.206316 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:00.196735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50015 (local address 127.0.98.62:40265)
0925 16:32:00.196931 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:00.196936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:00.199861 (+ 2925us) server_negotiation.cc:407] Connection header received
0925 16:32:00.200063 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:00.200066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:00.200142 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:00.200267 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:00.201440 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:00.202318 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:00.203837 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:00.204400 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:00.204579 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:00.205745 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:00.205749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:00.205751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:32:00.205813 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:00.205859 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:32:00.205875 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:00.205949 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:00.206051 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 16:32:00.206136 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 16:32:00.212075 22514 catalog_manager.cc:1347] Generated new cluster ID: 7f0c3ee0678f496d95171cc0c178484f
I20260925 16:32:00.212224 22514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:00.229146 22514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:00.230620 22514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:00.236591 22514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c66f88c91b944ae7981ae5be8e1e042b: Generated new TSK 0
I20260925 16:32:00.237690 22514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:32:00.246424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22423
2026-09-25T16:32:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:00 dist-test-slave-31kt krb5kdc[22522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:00 dist-test-slave-31kt krb5kdc[22522](info): set up 2 sockets
Sep 25 16:32:00 dist-test-slave-31kt krb5kdc[22522](info): commencing operation
krb5kdc: starting...
W20260925 16:32:01.242239 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:32:01 dist-test-slave-31kt krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:32:01.382213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:01.519590 22535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:01.519936 22535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:01.519984 22535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:01.524470 22535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:01.524566 22535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:01.524587 22535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:01.524603 22535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:01.524618 22535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:01.530483 22535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:01.531675 22535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:01.532740 22535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 16:32:01.542490 22535 server_base.cc:1034] running on GCE node
W20260925 16:32:01.542445 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:01.542443 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:01.542438 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:01.543586 22535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:01.544958 22535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:01.546291 22535 hybrid_clock.cc:648] HybridClock initialized: now 1790353921546275 us; error 49 us; skew 500 ppm
Sep 25 16:32:01 dist-test-slave-31kt krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:01.549868 22535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:01.551004 22535 webserver.cc:460] Webserver started at http://127.0.98.62:34579/ using document root <none> and password file <none>
I20260925 16:32:01.551616 22535 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:01.551690 22535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:01.551894 22535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:01.554221 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9"
format_stamp: "Formatted at 2026-09-25 16:32:01 on dist-test-slave-31kt"
server_key: "44a4c618fa8c4ef805682f5c9ce479e7"
server_key_iv: "c713c00788002cefff9a48bca43e2341"
server_key_version: "encryptionkey@0"
I20260925 16:32:01.554816 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9"
format_stamp: "Formatted at 2026-09-25 16:32:01 on dist-test-slave-31kt"
server_key: "44a4c618fa8c4ef805682f5c9ce479e7"
server_key_iv: "c713c00788002cefff9a48bca43e2341"
server_key_version: "encryptionkey@0"
I20260925 16:32:01.559361 22535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:01.562827 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:01.564546 22535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:01.564731 22535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9"
format_stamp: "Formatted at 2026-09-25 16:32:01 on dist-test-slave-31kt"
server_key: "44a4c618fa8c4ef805682f5c9ce479e7"
server_key_iv: "c713c00788002cefff9a48bca43e2341"
server_key_version: "encryptionkey@0"
I20260925 16:32:01.564869 22535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:01.571265 22535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:01.574885 22535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:01.575102 22535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:01.584342 22535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45035
I20260925 16:32:01.584365 22602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45035 every 8 connection(s)
I20260925 16:32:01.585695 22535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:01.590016 22603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:01.591768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22535
I20260925 16:32:01.591925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:01.592212 392 external_mini_cluster.cc:1428] Setting key 6e8eec32d0a664d22f420576b6ce53cd
I20260925 16:32:01.598222 22603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9: Bootstrap starting.
I20260925 16:32:01.599025 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:01.593990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34707 (local address 127.0.98.62:45035)
0925 16:32:01.594673 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:01.594692 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:01.594743 (+ 51us) server_negotiation.cc:407] Connection header received
0925 16:32:01.595581 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:01.595606 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:01.596171 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:01.596630 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:01.597593 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:01.597598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:01.597613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:32:01.597625 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:01.597686 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:32:01.597708 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:01.597886 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:01.597966 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 16:32:01.598161 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":208,"threads_started":1}
I20260925 16:32:01.601629 22603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:01.602875 22603 log.cc:824] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:01.603484 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:32:01.606217 22603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9: No bootstrap required, opened a new log
I20260925 16:32:01.608505 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:01.606367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40709 (local address 127.0.98.62:45035)
0925 16:32:01.606526 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:01.606531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:01.606722 (+ 191us) server_negotiation.cc:407] Connection header received
0925 16:32:01.606849 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:01.606853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:01.606902 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:01.607038 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:01.607924 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:01.607928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:01.607932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 16:32:01.607944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:01.607988 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 16:32:01.608006 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:01.608102 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:01.608299 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 16:32:01.608350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 16:32:01.611744 22603 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45035 } }
I20260925 16:32:01.612095 22603 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 16:32:01.612133 22603 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5ba6ce1c5e465a9908488ad53f1ca9, State: Initialized, Role: FOLLOWER
I20260925 16:32:01.612653 22603 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45035 } }
I20260925 16:32:01.612808 22603 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 16:32:01.612870 22603 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 16:32:01.612957 22603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:01.614311 22603 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45035 } }
I20260925 16:32:01.614768 22603 leader_election.cc:302] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5ba6ce1c5e465a9908488ad53f1ca9; no voters:
I20260925 16:32:01.615185 22603 leader_election.cc:288] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 16:32:01.615417 22613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:01.615793 22613 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [term 1 LEADER]: Becoming Leader. State: Replica: cb5ba6ce1c5e465a9908488ad53f1ca9, State: Running, Role: LEADER
I20260925 16:32:01.616210 22613 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45035 } }
I20260925 16:32:01.616770 22603 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:01.618387 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45035 } } }
I20260925 16:32:01.618577 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:01.618923 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5ba6ce1c5e465a9908488ad53f1ca9. Latest consensus state: current_term: 1 leader_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5ba6ce1c5e465a9908488ad53f1ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45035 } } }
I20260925 16:32:01.619028 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9 [sys.catalog]: This master's current role is: LEADER
W20260925 16:32:01.622669 22628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:01.622780 22628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:32:01.622867 22625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:01.623824 22625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:01.631503 22625 catalog_manager.cc:1347] Generated new cluster ID: 76301eef0f264543aeeffef65a58c6e6
I20260925 16:32:01.631628 22625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:01.659052 22625 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:01.659538 22625 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:01.674548 22625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5ba6ce1c5e465a9908488ad53f1ca9: Generated new TSK 0
I20260925 16:32:01.675652 22625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 16:32:01.687387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22535
2026-09-25T16:32:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1441 ms)
[----------] 18 tests from AuthTokenIssuingTest (25977 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:01 dist-test-slave-31kt krb5kdc[22634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:01 dist-test-slave-31kt krb5kdc[22634](info): set up 2 sockets
Sep 25 16:32:01 dist-test-slave-31kt krb5kdc[22634](info): commencing operation
krb5kdc: starting...
W20260925 16:32:02.682374 392 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:32:02 dist-test-slave-31kt krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:32:02.822903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37069,127.0.98.61:38881,127.0.98.60:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:02.957617 22647 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:32:02.958112 22647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:02.958150 22647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:02.958215 22647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:02.962824 22647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:02.962947 22647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:02.962989 22647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:02.963014 22647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:02.963037 22647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:02.970098 22647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37069,127.0.98.61:38881,127.0.98.60:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:02.971730 22647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:02.973109 22647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:02.984586 22655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:02.984556 22647 server_base.cc:1034] running on GCE node
W20260925 16:32:02.984839 22653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:02.984505 22652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:02.985795 22647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:02.987365 22647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:02.988572 22647 hybrid_clock.cc:648] HybridClock initialized: now 1790353922988550 us; error 46 us; skew 500 ppm
Sep 25 16:32:02 dist-test-slave-31kt krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:02.992384 22647 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:02.993868 22647 webserver.cc:460] Webserver started at http://127.0.98.62:35297/ using document root <none> and password file <none>
I20260925 16:32:02.994711 22647 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:02.994812 22647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:02.995081 22647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:02.998165 22647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "bc9e84d099cf442a839dd4fe859f68b6"
format_stamp: "Formatted at 2026-09-25 16:32:02 on dist-test-slave-31kt"
server_key: "741ecc4e95cb7ffcc38e9a36e3ff49d8"
server_key_iv: "71176c69fb7c35486e4dba7e3ea25d9c"
server_key_version: "encryptionkey@0"
I20260925 16:32:02.999264 22647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc9e84d099cf442a839dd4fe859f68b6"
format_stamp: "Formatted at 2026-09-25 16:32:02 on dist-test-slave-31kt"
server_key: "741ecc4e95cb7ffcc38e9a36e3ff49d8"
server_key_iv: "71176c69fb7c35486e4dba7e3ea25d9c"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.005406 22647 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 16:32:03.009976 22662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:03.012315 22647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:03.012578 22647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "bc9e84d099cf442a839dd4fe859f68b6"
format_stamp: "Formatted at 2026-09-25 16:32:02 on dist-test-slave-31kt"
server_key: "741ecc4e95cb7ffcc38e9a36e3ff49d8"
server_key_iv: "71176c69fb7c35486e4dba7e3ea25d9c"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.012765 22647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:03.037324 22647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:03.042379 22647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:03.042692 22647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:03.054486 22647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37069
I20260925 16:32:03.054822 22714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37069 every 8 connection(s)
I20260925 16:32:03.056420 22647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:03.061460 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:03.065968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22647
I20260925 16:32:03.066179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:03.066488 392 external_mini_cluster.cc:1428] Setting key 5e34e664bfe155d6e9a4b01cc9d563f2
I20260925 16:32:03.068475 22715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:03.088510 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.071787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37069 (local address 127.0.0.1:58148)
0925 16:32:03.072528 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.072575 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.073568 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.074352 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.074358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.074864 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.075399 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.075437 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.077128 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.077135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.078192 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.078213 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.079194 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.079203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.079364 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.080180 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.080204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.082610 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.085232 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.085242 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.085254 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.085545 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.085911 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.085914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.085916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.086034 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.086691 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.086697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.086831 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.086993 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.087389 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":99,"spinlock_wait_cycles":22400,"thread_start_us":221,"threads_started":1}
I20260925 16:32:03.088510 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.071785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.0.98.62:37069)
0925 16:32:03.072385 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.072406 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.072699 (+ 293us) server_negotiation.cc:407] Connection header received
0925 16:32:03.073937 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.073942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.074153 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.074338 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.075677 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.076849 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.078401 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.078858 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.079036 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.082811 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.082818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.082832 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.082859 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.085086 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.085661 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.085667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.085668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.085825 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.086125 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.086128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.086130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.086575 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.086703 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.087066 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.087327 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":159,"threads_started":1}
I20260925 16:32:03.092355 22715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:03.093400 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.093047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38026)
0925 16:32:03.093233 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.093319 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 16:32:03.093616 22666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
I20260925 16:32:03.094558 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.094342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38034)
0925 16:32:03.094448 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.094496 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 16:32:03.094692 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38881: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 16:32:03.101082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38881
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37069,127.0.98.61:38881,127.0.98.60:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:32:03.144030 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } attempt: 1
I20260925 16:32:03.145499 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.145008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38050)
0925 16:32:03.145270 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.145363 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 16:32:03.146889 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.146347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38052)
0925 16:32:03.146492 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.146566 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 16:32:03.147221 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38881: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
I20260925 16:32:03.215417 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } attempt: 2
I20260925 16:32:03.216979 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.216245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38062)
0925 16:32:03.216554 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.216802 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 16:32:03.218283 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.217875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38078)
0925 16:32:03.218059 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.218165 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:32:03.218533 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38881: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
W20260925 16:32:03.239998 22722 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:32:03.240373 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:03.240419 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:03.240487 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:03.244972 22722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:03.245074 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:03.245102 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:03.245124 22722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:03.245146 22722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:03.252080 22722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37069,127.0.98.61:38881,127.0.98.60:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:03.253407 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:03.254634 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:03.265857 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:03.265969 22722 server_base.cc:1034] running on GCE node
W20260925 16:32:03.265857 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:03.265857 22730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:03.267185 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:03.268895 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:03.270166 22722 hybrid_clock.cc:648] HybridClock initialized: now 1790353923270136 us; error 59 us; skew 500 ppm
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:03.274294 22722 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 16:32:03.275832 22722 webserver.cc:460] Webserver started at http://127.0.98.61:36341/ using document root <none> and password file <none>
I20260925 16:32:03.276624 22722 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:03.276721 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:03.277006 22722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:03.279950 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data/instance:
uuid: "b91c0bcc584646f69a37e863ecfa3a28"
format_stamp: "Formatted at 2026-09-25 16:32:03 on dist-test-slave-31kt"
server_key: "40ad610f36df284d1aa66fd9e95f7df5"
server_key_iv: "dd17fa320e279393782ab61c9919afbb"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.280706 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal/instance:
uuid: "b91c0bcc584646f69a37e863ecfa3a28"
format_stamp: "Formatted at 2026-09-25 16:32:03 on dist-test-slave-31kt"
server_key: "40ad610f36df284d1aa66fd9e95f7df5"
server_key_iv: "dd17fa320e279393782ab61c9919afbb"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.286273 22722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 16:32:03.290318 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:03.292362 22722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 16:32:03.292596 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal
uuid: "b91c0bcc584646f69a37e863ecfa3a28"
format_stamp: "Formatted at 2026-09-25 16:32:03 on dist-test-slave-31kt"
server_key: "40ad610f36df284d1aa66fd9e95f7df5"
server_key_iv: "dd17fa320e279393782ab61c9919afbb"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.292778 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:03.309891 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } attempt: 3
I20260925 16:32:03.311213 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.310722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38094)
0925 16:32:03.310972 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.311107 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 16:32:03.312410 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.312013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38104)
0925 16:32:03.312256 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.312331 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260925 16:32:03.312554 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38881: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38881: connect: Connection refused (error 111)
I20260925 16:32:03.317794 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:03.322763 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:03.323041 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:03.333015 22722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38881
I20260925 16:32:03.333525 22789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38881 every 8 connection(s)
I20260925 16:32:03.334556 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:32:03.339133 22790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:03.341548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22722
I20260925 16:32:03.341801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-1/wal/instance
I20260925 16:32:03.342221 392 external_mini_cluster.cc:1428] Setting key 6a874b251cf50267308c45f3c37557df
I20260925 16:32:03.346752 22790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:03.370540 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.349916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.0.98.62:37069)
0925 16:32:03.350069 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.350072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.352483 (+ 2411us) server_negotiation.cc:407] Connection header received
0925 16:32:03.353920 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.353927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.354038 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.354279 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.356071 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.356806 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.358676 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.359215 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.359391 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.364468 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.364473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.364476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.364532 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.366327 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.367571 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.367576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.367577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.367720 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.368526 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.368530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.368532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.368943 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.369078 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.370102 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.370350 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 16:32:03.371615 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.351095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37069 (local address 127.0.0.1:58158)
0925 16:32:03.352226 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.352309 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.353452 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.354544 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.354568 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.355086 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.355795 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.355818 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.357080 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.357085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.358316 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.358336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.359593 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.359599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.359931 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.360894 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.360920 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.363844 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.366570 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.366585 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.366621 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.367244 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.367853 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.367858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.367861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.368189 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.369179 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.369188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.369677 (+ 489us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.369965 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.370359 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":321,"threads_started":1}
I20260925 16:32:03.373709 22790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:32:03.382653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44293
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37069,127.0.98.61:38881,127.0.98.60:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:32:03.388561 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.374424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38108)
0925 16:32:03.374568 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.374585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.374680 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.375715 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.375718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.375998 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.376101 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.376108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.377872 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.377876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.378557 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.378570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.379607 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.379611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.379927 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.380669 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.380682 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.382761 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.386352 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.386356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.386358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.386608 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.387007 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.387009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.387011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.387104 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.387891 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.387896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.388073 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.388229 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.388409 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 16:32:03.388885 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.374667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.0.98.61:38881)
0925 16:32:03.375275 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.375287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.375316 (+ 29us) server_negotiation.cc:407] Connection header received
0925 16:32:03.375361 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.375366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.375574 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.375718 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.376220 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.377612 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.378777 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.379452 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.379573 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.383254 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.383280 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.383305 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.383342 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.386016 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.386756 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.386765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.386767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.386879 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.387200 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.387203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.387205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.387657 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.387831 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.388401 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.388636 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":210,"threads_started":1}
I20260925 16:32:03.391626 22790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:03.392735 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.392306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36140)
0925 16:32:03.392521 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.392622 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 16:32:03.393083 22740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
I20260925 16:32:03.394641 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.394262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36148)
0925 16:32:03.394472 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.394547 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260925 16:32:03.394886 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
I20260925 16:32:03.444324 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 1
I20260925 16:32:03.445498 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.445180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36152)
0925 16:32:03.445342 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.445407 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 16:32:03.446695 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.446383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36158)
0925 16:32:03.446550 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.446617 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 16:32:03.446910 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
I20260925 16:32:03.455791 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } attempt: 4
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:32:03.468456 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.456651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38122)
0925 16:32:03.457046 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.457064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.457193 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.457627 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.457630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.458057 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.458277 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.458288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.459513 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.459518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.460156 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.460172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.460997 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.461002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.461160 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.462020 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.462039 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.463525 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.466063 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.466067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.466069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.466519 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.467025 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.467028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.467030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.467153 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.467824 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.467827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.467996 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.468119 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.468297 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260925 16:32:03.468571 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.456902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.61:38881)
0925 16:32:03.457113 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.457119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.457144 (+ 25us) server_negotiation.cc:407] Connection header received
0925 16:32:03.457325 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.457328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.457404 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.457520 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.458431 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.459296 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.460333 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.460775 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.460898 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.463773 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.463779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.463782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.463829 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.465848 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.466652 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.466657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.466659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.466780 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.467315 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.467319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.467321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.467674 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.467797 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.468196 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.468375 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 16:32:03.469672 22715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:03.470484 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.470215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36160)
0925 16:32:03.470338 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.470411 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 16:32:03.472020 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.471737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36162)
0925 16:32:03.471881 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.471942 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 16:32:03.472182 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
I20260925 16:32:03.515179 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 2
I20260925 16:32:03.516417 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.516013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36172)
0925 16:32:03.516216 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.516305 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 16:32:03.517689 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.517207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36188)
0925 16:32:03.517488 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.517586 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260925 16:32:03.517935 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
W20260925 16:32:03.530542 22796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:32:03.530974 22796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:03.531001 22796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:03.531047 22796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 16:32:03.531380 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 1
I20260925 16:32:03.532245 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.531951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36200)
0925 16:32:03.532102 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.532164 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 16:32:03.533327 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.533036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36202)
0925 16:32:03.533194 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.533261 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 16:32:03.533490 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
W20260925 16:32:03.536420 22796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:03.536541 22796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:03.536562 22796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:03.536580 22796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:03.536597 22796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:03.543536 22796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37069,127.0.98.61:38881,127.0.98.60:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:03.545147 22796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:03.546643 22796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:03.557528 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:03.557528 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:03.557548 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:03.557936 22796 server_base.cc:1034] running on GCE node
I20260925 16:32:03.558832 22796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:03.560405 22796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:03.561621 22796 hybrid_clock.cc:648] HybridClock initialized: now 1790353923561602 us; error 47 us; skew 500 ppm
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:03.565704 22796 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 16:32:03.567180 22796 webserver.cc:460] Webserver started at http://127.0.98.60:35647/ using document root <none> and password file <none>
I20260925 16:32:03.567953 22796 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:03.568040 22796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:03.568293 22796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:03.571241 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data/instance:
uuid: "2416f4037cf74647ac472c5402b318bd"
format_stamp: "Formatted at 2026-09-25 16:32:03 on dist-test-slave-31kt"
server_key: "31691d8a8d8c4c43a0009feb72ac25be"
server_key_iv: "bd550a04163bd891d69039ca4dcf694e"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.571993 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal/instance:
uuid: "2416f4037cf74647ac472c5402b318bd"
format_stamp: "Formatted at 2026-09-25 16:32:03 on dist-test-slave-31kt"
server_key: "31691d8a8d8c4c43a0009feb72ac25be"
server_key_iv: "bd550a04163bd891d69039ca4dcf694e"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.577509 22796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 16:32:03.581696 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:03.583565 22796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 16:32:03.583762 22796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal
uuid: "2416f4037cf74647ac472c5402b318bd"
format_stamp: "Formatted at 2026-09-25 16:32:03 on dist-test-slave-31kt"
server_key: "31691d8a8d8c4c43a0009feb72ac25be"
server_key_iv: "bd550a04163bd891d69039ca4dcf694e"
server_key_version: "encryptionkey@0"
I20260925 16:32:03.583928 22796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:03.600677 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 2
I20260925 16:32:03.601550 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.601253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36216)
0925 16:32:03.601399 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.601468 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 16:32:03.602628 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.602349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36226)
0925 16:32:03.602502 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.602563 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 16:32:03.602809 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
I20260925 16:32:03.603797 22796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:03.607923 22796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:03.608155 22796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:03.609153 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 3
I20260925 16:32:03.610282 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.609736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36238)
0925 16:32:03.610013 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.610117 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 16:32:03.611311 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.610910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36254)
0925 16:32:03.611131 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.611212 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260925 16:32:03.611554 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44293: connect: Connection refused (error 111)
I20260925 16:32:03.618113 22796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44293
I20260925 16:32:03.618464 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44293 every 8 connection(s)
I20260925 16:32:03.619531 22796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:32:03.622814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 22796
I20260925 16:32:03.622974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790353612433705-392-0/minicluster-data/master-2/wal/instance
I20260925 16:32:03.623320 392 external_mini_cluster.cc:1428] Setting key 1b4337a0a7a666698a2ab5c158860f94
I20260925 16:32:03.623863 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:03.630074 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:03.643695 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.632357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.0.98.62:37069)
0925 16:32:03.632576 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.632581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.632819 (+ 238us) server_negotiation.cc:407] Connection header received
0925 16:32:03.633034 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.633037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.633112 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.633228 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.634461 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.635078 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.636471 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.636871 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.636982 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.640246 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.640250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.640252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.640287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.641694 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.642307 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.642311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.642313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.642393 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.642658 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.642661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.642663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.642878 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.642975 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.643380 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.643519 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 16:32:03.643646 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.625329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.0.98.60:44293)
0925 16:32:03.626126 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.626149 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.626215 (+ 66us) server_negotiation.cc:407] Connection header received
0925 16:32:03.627533 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.627570 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.628251 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.628873 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.629997 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.631368 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.632540 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.633428 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.633578 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.637622 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.637631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.637650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.637696 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.640046 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.640608 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.640615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.640622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.640750 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.641017 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.641021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.641022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.641601 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.641865 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.642170 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.642584 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":205,"threads_started":1}
I20260925 16:32:03.643854 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.632215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37069 (local address 127.0.0.1:58160)
0925 16:32:03.632653 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.632715 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.632907 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.633358 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.633364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.633985 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.634217 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.634226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.635275 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.635279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.636275 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.636290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.637091 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.637097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.637244 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.637955 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.637971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.640017 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.641877 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.641890 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.641898 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.642201 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.642488 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.642491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.642493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.642574 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.642988 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.642997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.643141 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.643293 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.643540 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
I20260925 16:32:03.646147 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:03.657868 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.646793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38881 (local address 127.0.0.1:38138)
0925 16:32:03.646992 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.647013 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.647133 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.647566 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.647569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.647808 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.647973 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.647981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.649402 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.649406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.650333 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.650347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.651300 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.651305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.651419 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.651924 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.651937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.653579 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.655937 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.655942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.655944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.656179 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.656532 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.656536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.656538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.656646 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.657198 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.657201 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.657328 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.657444 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.657628 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 16:32:03.657965 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.646935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.61:38881)
0925 16:32:03.647075 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.647080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.647095 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:03.647272 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.647276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.647343 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.647398 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.648149 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.649208 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.650597 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.650990 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.651111 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.653907 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.653912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.653916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.653954 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.655773 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.656302 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.656307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.656309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.656412 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.656778 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.656782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.656784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.657067 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.657158 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.657552 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.657714 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 16:32:03.658674 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.647129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.98.62:37069)
0925 16:32:03.647258 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.647262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.647302 (+ 40us) server_negotiation.cc:407] Connection header received
0925 16:32:03.647441 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.647443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.647489 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.647564 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.648378 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.649239 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.650550 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.650912 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.651023 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.654495 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.654498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.654500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.654530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.655847 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.656394 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.656398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.656399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.656511 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.656897 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.656900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.656901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.657158 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.657440 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.657632 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.657804 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 16:32:03.659061 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:32:03.669615 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.659770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36280)
0925 16:32:03.659955 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.659973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.660074 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.660530 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.660534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.660756 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.660888 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.660895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.661714 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.661718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.662493 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.662503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.663049 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.663053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.663147 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.663756 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.663768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.665410 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.667555 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.667560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.667562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.667841 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.668231 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.668234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.668235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.668435 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.668993 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.668996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.669120 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.669234 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.669415 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 16:32:03.670008 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.660406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.98.61:38881)
0925 16:32:03.660561 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.660565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.660580 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:03.660706 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.660708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.660755 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.660825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.661525 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.662152 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.663095 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.663338 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.663429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.666224 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.666229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.666232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.666267 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.667844 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.668452 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.668456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.668458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.668561 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.668858 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.668860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.668861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.669130 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.669416 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.669649 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.669843 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 16:32:03.669869 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.659934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.0.98.60:44293)
0925 16:32:03.660060 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.660063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.660077 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:32:03.660194 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.660197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.660252 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.660340 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.660991 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.661570 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.662646 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.662912 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.663018 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.665712 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.665717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.665721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.665791 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.667398 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.667987 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.667991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.667993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.668119 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.668564 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.668567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.668569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.668871 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.668952 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.669383 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.669556 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 16:32:03.674822 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: Bootstrap starting.
I20260925 16:32:03.675420 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:32:03.678391 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:03.677629 22683 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc9e84d099cf442a839dd4fe859f68b6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174
I20260925 16:32:03.679299 22865 log.cc:824] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:03.683146 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: No bootstrap required, opened a new log
I20260925 16:32:03.688359 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.688922 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2416f4037cf74647ac472c5402b318bd, State: Initialized, Role: FOLLOWER
I20260925 16:32:03.689419 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.690187 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.690418 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:03.691088 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: configured and running, proceeding with master startup.
W20260925 16:32:03.698089 22882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:03.698272 22882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:32:03.696336 22796 master_runner.cc:186] Error getting master registration for 127.0.98.62:37069: OK, instance_id { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" instance_seqno: 1790353923051245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:32:03.701877 22796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:32:03.703059 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 3
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:32:03.716271 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.703719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36296)
0925 16:32:03.703972 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.703992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.704097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.704535 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.704538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.704782 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.704954 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.704963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.706241 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.706244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.707406 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.707430 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.708193 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.708196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.708311 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.709245 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.709269 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.710917 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.713716 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.713722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.713725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.714322 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.714820 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.714823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.714826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.714959 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.715583 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.715586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.715745 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.715863 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.716067 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 16:32:03.716460 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.703930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.0.98.60:44293)
0925 16:32:03.704138 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.704144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.704168 (+ 24us) server_negotiation.cc:407] Connection header received
0925 16:32:03.704245 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.704250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.704324 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.704491 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.705126 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.706076 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.707620 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.708025 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.708169 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.711226 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.711232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.711236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.711295 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.713482 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.714540 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.714548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.714550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.714690 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.715070 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.715073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.715075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.715472 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.715593 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.715950 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.716143 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 16:32:03.721383 22715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: Bootstrap starting.
I20260925 16:32:03.724881 22715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:03.726254 22715 log.cc:824] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:03.729803 22715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: No bootstrap required, opened a new log
I20260925 16:32:03.733665 22715 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.734026 22715 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9e84d099cf442a839dd4fe859f68b6, State: Initialized, Role: FOLLOWER
I20260925 16:32:03.734404 22715 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.735057 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.735327 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:03.736027 22715 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:03.736126 22683 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 16:32:03.736225 22683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:03.737733 22683 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.738934 22683 leader_election.cc:288] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [CANDIDATE]: Term 1 election: Requested vote from peers b91c0bcc584646f69a37e863ecfa3a28 (127.0.98.61:38881), 2416f4037cf74647ac472c5402b318bd (127.0.98.60:44293)
I20260925 16:32:03.739666 22839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc9e84d099cf442a839dd4fe859f68b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2416f4037cf74647ac472c5402b318bd"
I20260925 16:32:03.739974 22839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:03.741729 22839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9e84d099cf442a839dd4fe859f68b6 in term 1.
I20260925 16:32:03.743201 22665 leader_election.cc:302] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2416f4037cf74647ac472c5402b318bd, bc9e84d099cf442a839dd4fe859f68b6; no voters:
W20260925 16:32:03.743129 22896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:03.744683 22896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:32:03.744023 22885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:03.745236 22885 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [term 1 LEADER]: Becoming Leader. State: Replica: bc9e84d099cf442a839dd4fe859f68b6, State: Running, Role: LEADER
I20260925 16:32:03.745712 22885 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.750200 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9e84d099cf442a839dd4fe859f68b6. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.750419 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:03.757804 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } attempt: 4
I20260925 16:32:03.754509 22647 master_runner.cc:186] Error getting master registration for 127.0.98.61:38881: OK, instance_id { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" instance_seqno: 1790353923330144 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:32:03.760624 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.747837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.62:37069)
0925 16:32:03.747968 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.747973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.749114 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 16:32:03.749341 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.749345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.749423 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.749482 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.750781 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.751658 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.753047 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.753351 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.753428 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.755837 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.755842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.755845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.755880 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.757556 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.758995 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.759000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.759001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.759122 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.759477 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.759480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.759482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.759772 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.759910 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.760273 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.760433 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:32:03.761152 22647 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:32:03.763386 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.747222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.0.98.60:44293)
0925 16:32:03.747386 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.747391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.752213 (+ 4822us) server_negotiation.cc:407] Connection header received
0925 16:32:03.752563 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.752567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.752664 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.752763 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.755013 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.755847 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.757080 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.757435 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.757521 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.759511 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.759515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.759519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.759553 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.761331 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.761912 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.761916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.761919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.762009 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.762288 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.762292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.762294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.762537 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.762676 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.763034 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.763187 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 16:32:03 dist-test-slave-31kt krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:32:03.770437 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.748417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.0.98.61:38881)
0925 16:32:03.748586 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.748591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.754110 (+ 5519us) server_negotiation.cc:407] Connection header received
0925 16:32:03.754300 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.754303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.754363 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.754416 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.762182 (+ 7766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.763017 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.764302 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.764585 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.764658 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.766644 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.766648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.766651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.766680 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.768481 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.768999 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.769003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.769005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.769090 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.769385 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.769388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.769390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.769615 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.769741 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.770128 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.770254 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 16:32:03.774287 22907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:03.774354 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.759396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.0.98.60:44293)
0925 16:32:03.763614 (+ 4218us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.763618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.763633 (+ 15us) server_negotiation.cc:407] Connection header received
0925 16:32:03.763670 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.763673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.763721 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.763803 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.764529 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.765342 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.766759 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.767120 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.767221 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.769660 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:03.769664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:03.769667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:03.769719 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:03.772534 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.773011 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.773015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.773017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.773121 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:03.773397 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:03.773400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:03.773402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:03.773671 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:03.773785 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:03.774091 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 16:32:03.774206 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4145,"thread_start_us":179,"threads_started":1}
I20260925 16:32:03.774354 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:03.758541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44293 (local address 127.0.0.1:36312)
0925 16:32:03.759093 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:03.759110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:03.759199 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:03.763851 (+ 4652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:03.763854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:03.764096 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:03.764224 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.764231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.765791 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.765795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.766614 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:03.766623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:03.767389 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:03.767393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:03.767480 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:03.767873 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:03.767889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:03.769471 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:03.772667 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.772670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.772673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.772918 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.773203 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:03.773206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:03.773207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:03.773307 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:03.773785 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:03.773788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:03.773889 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 16:32:03.773980 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 16:32:03.774168 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424}
I20260925 16:32:03.775595 22907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:03.778894 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: Bootstrap starting.
I20260925 16:32:03.781584 22790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:03.785104 22790 log.cc:824] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:03.785496 22839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [term 1 FOLLOWER]: Refusing update from remote peer bc9e84d099cf442a839dd4fe859f68b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:03.786311 22885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:03.788805 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: No bootstrap required, opened a new log
I20260925 16:32:03.791745 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9e84d099cf442a839dd4fe859f68b6. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.791980 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:32:03.795845 22668 tablet.cc:2369] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 16:32:03.795801 22790 raft_consensus.cc:348] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.796286 22790 raft_consensus.cc:729] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91c0bcc584646f69a37e863ecfa3a28, State: Initialized, Role: FOLLOWER
I20260925 16:32:03.796710 22790 consensus_queue.cc:260] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } }
I20260925 16:32:03.797385 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.797695 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:03.800926 22764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc9e84d099cf442a839dd4fe859f68b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b91c0bcc584646f69a37e863ecfa3a28"
I20260925 16:32:03.801170 22763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:03.800901 22790 sys_catalog.cc:564] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:03.803095 22763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [term 1 FOLLOWER]: Refusing update from remote peer bc9e84d099cf442a839dd4fe859f68b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:03.806502 22885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:03.805697 22764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9e84d099cf442a839dd4fe859f68b6 in term 1.
I20260925 16:32:03.813258 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9e84d099cf442a839dd4fe859f68b6. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.814875 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:03.815071 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.815240 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:03.817771 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.817989 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:03.818257 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.818423 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:03.819288 22909 mvcc.cc:205] Tried to move back new op lower bound from 7333289671815118848 to 7333289671669768192. Current Snapshot: MvccSnapshot[applied={T|T < 7333289671815118848}]
I20260925 16:32:03.821554 22911 mvcc.cc:205] Tried to move back new op lower bound from 7333289671815118848 to 7333289671669768192. Current Snapshot: MvccSnapshot[applied={T|T < 7333289671815118848}]
I20260925 16:32:03.822751 22924 mvcc.cc:205] Tried to move back new op lower bound from 7333289671815118848 to 7333289671669768192. Current Snapshot: MvccSnapshot[applied={T|T < 7333289671815118848}]
I20260925 16:32:03.824116 22907 catalog_manager.cc:1347] Generated new cluster ID: 2d5f2f7c69354f4484cc7bb24ea248fa
I20260925 16:32:03.824231 22907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:03.824899 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc9e84d099cf442a839dd4fe859f68b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9e84d099cf442a839dd4fe859f68b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } peers { permanent_uuid: "b91c0bcc584646f69a37e863ecfa3a28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38881 } } peers { permanent_uuid: "2416f4037cf74647ac472c5402b318bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44293 } } }
I20260925 16:32:03.825089 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 16:32:03.829002 22926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:03.829110 22926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:32:03.853314 22907 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:03.855072 22907 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:03.881116 22907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc9e84d099cf442a839dd4fe859f68b6: Generated new TSK 0
I20260925 16:32:03.882620 22907 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:32:03.926766 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.923808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.0.98.60:44293)
0925 16:32:03.923988 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.924005 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.925327 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 16:32:03.925511 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.925515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.925589 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.925729 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.926584 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36326: BlockingRecv error: recv got EOF from 127.0.0.1:36326 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 16:32:03.927058 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.924829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.0.98.61:38881)
0925 16:32:03.924963 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.924968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.925731 (+ 763us) server_negotiation.cc:407] Connection header received
0925 16:32:03.925989 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.925994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.926061 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.926119 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.926860 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38150: BlockingRecv error: recv got EOF from 127.0.0.1:38150 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 16:32:03.928691 22718 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.924471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.0.98.62:37069)
0925 16:32:03.924603 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.924607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.927662 (+ 3055us) server_negotiation.cc:407] Connection header received
0925 16:32:03.927748 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.927751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.927823 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.927932 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.928529 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58196: BlockingRecv error: recv got EOF from 127.0.0.1:58196 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 16:32:03.932922 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.931428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.0.98.61:38881)
0925 16:32:03.931593 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.931599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.931720 (+ 121us) server_negotiation.cc:407] Connection header received
0925 16:32:03.931958 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.931963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.932085 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.932222 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.932748 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38160: BlockingRecv error: recv got EOF from 127.0.0.1:38160 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 16:32:03.938198 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.936492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.0.98.61:38881)
0925 16:32:03.936676 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.936682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.936703 (+ 21us) server_negotiation.cc:407] Connection header received
0925 16:32:03.936808 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.936812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.936885 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.937006 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.938028 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38170: BlockingRecv error: recv got EOF from 127.0.0.1:38170 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 16:32:03.945741 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.943828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.98.61:38881)
0925 16:32:03.944041 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.944048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.944569 (+ 521us) server_negotiation.cc:407] Connection header received
0925 16:32:03.944756 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.944760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.944843 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.944956 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.945610 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38178: BlockingRecv error: recv got EOF from 127.0.0.1:38178 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 16:32:03.954346 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.952680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.0.98.61:38881)
0925 16:32:03.952872 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.952877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.953129 (+ 252us) server_negotiation.cc:407] Connection header received
0925 16:32:03.953312 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.953316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.953399 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.953551 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.954178 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38192: BlockingRecv error: recv got EOF from 127.0.0.1:38192 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 16:32:03.966445 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.964916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.0.98.61:38881)
0925 16:32:03.965101 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.965107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.965302 (+ 195us) server_negotiation.cc:407] Connection header received
0925 16:32:03.965475 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.965479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.965552 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.965688 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.966276 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38202: BlockingRecv error: recv got EOF from 127.0.0.1:38202 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 16:32:03.986423 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:03.984846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.0.98.61:38881)
0925 16:32:03.985043 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:03.985048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:03.985265 (+ 217us) server_negotiation.cc:407] Connection header received
0925 16:32:03.985452 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:03.985456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:03.985528 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:03.985659 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:03.986265 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38214: BlockingRecv error: recv got EOF from 127.0.0.1:38214 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 16:32:04.022579 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:04.020863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.61:38881)
0925 16:32:04.021072 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:04.021078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:04.021443 (+ 365us) server_negotiation.cc:407] Connection header received
0925 16:32:04.021657 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:04.021661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:04.021724 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:04.021876 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:04.022434 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38220: BlockingRecv error: recv got EOF from 127.0.0.1:38220 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 16:32:04.091051 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:04.089475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.98.61:38881)
0925 16:32:04.089632 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:04.089638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:04.089881 (+ 243us) server_negotiation.cc:407] Connection header received
0925 16:32:04.090037 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:04.090040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:04.090107 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:04.090233 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:04.090894 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38232: BlockingRecv error: recv got EOF from 127.0.0.1:38232 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 16:32:04.223764 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:04.222215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.98.61:38881)
0925 16:32:04.222400 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:04.222406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:04.222583 (+ 177us) server_negotiation.cc:407] Connection header received
0925 16:32:04.222734 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:04.222739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:04.222816 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:04.222971 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:04.223597 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38240: BlockingRecv error: recv got EOF from 127.0.0.1:38240 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 16:32:04.483595 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:04.482581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.98.61:38881)
0925 16:32:04.482784 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:04.482789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:04.482803 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:32:04.482859 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:04.482861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:04.482921 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:04.482991 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:04.483483 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38248: BlockingRecv error: recv got EOF from 127.0.0.1:38248 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 16:32:04.700280 22882 catalog_manager.cc:1259] Loaded cluster ID: 2d5f2f7c69354f4484cc7bb24ea248fa
I20260925 16:32:04.700395 22882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: loading cluster ID for follower catalog manager: success
I20260925 16:32:04.703220 22882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: acquiring CA information for follower catalog manager: success
I20260925 16:32:04.704829 22882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2416f4037cf74647ac472c5402b318bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:32:04.830657 22926 catalog_manager.cc:1259] Loaded cluster ID: 2d5f2f7c69354f4484cc7bb24ea248fa
I20260925 16:32:04.830760 22926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: loading cluster ID for follower catalog manager: success
I20260925 16:32:04.832728 22926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: acquiring CA information for follower catalog manager: success
I20260925 16:32:04.834653 22926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b91c0bcc584646f69a37e863ecfa3a28: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:32:05.005460 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:04.998982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.98.61:38881)
0925 16:32:04.999241 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:04.999245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:04.999392 (+ 147us) server_negotiation.cc:407] Connection header received
0925 16:32:04.999510 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:04.999513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:04.999685 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:04.999820 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:05.000378 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.001147 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.002685 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.003072 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.003215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:05.003794 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:05.004759 (+ 965us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:05.004971 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:05.005133 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 16:32:05.005241 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260925 16:32:05.013417 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:05.009442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.0.98.60:44293)
0925 16:32:05.009844 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:05.009848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:05.009862 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:32:05.009955 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:05.009958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:05.010096 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:05.010209 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:05.010449 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.010986 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.011863 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.012097 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.012186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:05.012505 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:05.012997 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:05.013121 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:05.013181 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 16:32:05.013245 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260925 16:32:05.022320 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:05.017583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.98.60:44293)
0925 16:32:05.017735 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:05.017740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:05.017980 (+ 240us) server_negotiation.cc:407] Connection header received
0925 16:32:05.018249 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:05.018253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:05.018329 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:05.018407 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:05.018751 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.019637 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.020804 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.021044 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.021137 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:05.021483 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:05.021602 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:05.021676 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:05.022078 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 16:32:05.022162 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 16:32:05.022715 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:05.017878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.61:38881)
0925 16:32:05.018071 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:05.018076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:05.018424 (+ 348us) server_negotiation.cc:407] Connection header received
0925 16:32:05.018506 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:05.018509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:05.018579 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:05.018653 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:05.019030 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.019668 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.021305 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.021662 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.021842 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:05.022143 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:05.022240 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:05.022284 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:05.022455 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 16:32:05.022543 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 16:32:05.023744 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:05.017425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.0.98.62:37069)
0925 16:32:05.017727 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:05.017732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:05.018106 (+ 374us) server_negotiation.cc:407] Connection header received
0925 16:32:05.018236 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:05.018240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:05.018390 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:05.018514 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:05.019019 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.019830 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.021862 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:05.022209 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:05.022310 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:05.022751 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:05.023153 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:05.023236 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:05.023499 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 16:32:05.023588 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260925 16:32:05.028399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22647
I20260925 16:32:05.035946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22722
I20260925 16:32:05.043519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 22796
2026-09-25T16:32:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3357 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 16:32:05 dist-test-slave-31kt krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 16:32:05 dist-test-slave-31kt krb5kdc[23012](info): set up 2 sockets
Sep 25 16:32:05 dist-test-slave-31kt krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260925 16:32:06.018520 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/test-user.keytab.
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T16:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T16:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 16:32:06.161558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42733
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42733,127.0.98.61:36115,127.0.98.60:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 16:32:06.291776 23025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:32:06.292533 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:06.292614 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:06.292680 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:06.297401 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:06.297528 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:06.297559 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:06.297582 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:06.297603 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:06.305171 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42733,127.0.98.61:36115,127.0.98.60:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:06.306537 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:06.307693 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:06.317832 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:06.317816 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:06.317816 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:06.318372 23025 server_base.cc:1034] running on GCE node
I20260925 16:32:06.319180 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:06.320766 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:06.321955 23025 hybrid_clock.cc:648] HybridClock initialized: now 1790353926321932 us; error 53 us; skew 500 ppm
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:06.333235 23025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 16:32:06.334817 23025 webserver.cc:460] Webserver started at http://127.0.98.62:40417/ using document root <none> and password file <none>
I20260925 16:32:06.335613 23025 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:06.335708 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:06.335983 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:06.338861 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data/instance:
uuid: "6a1f94304e7a4af5b42921953ec11c28"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "a968cc3135cfd065c22e4ec3d2ade9d4"
server_key_iv: "4116cff90fca1fb7120a4b09379c10d8"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.339751 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a1f94304e7a4af5b42921953ec11c28"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "a968cc3135cfd065c22e4ec3d2ade9d4"
server_key_iv: "4116cff90fca1fb7120a4b09379c10d8"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.344935 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 16:32:06.348567 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:06.350736 23025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 16:32:06.350930 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal
uuid: "6a1f94304e7a4af5b42921953ec11c28"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "a968cc3135cfd065c22e4ec3d2ade9d4"
server_key_iv: "4116cff90fca1fb7120a4b09379c10d8"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.351107 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:06.364562 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:06.369557 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:06.369875 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:06.379328 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42733
I20260925 16:32:06.379652 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42733 every 8 connection(s)
I20260925 16:32:06.380641 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/data/info.pb
I20260925 16:32:06.384101 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:06.388569 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:06.390636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 23025
I20260925 16:32:06.390758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-0/wal/instance
I20260925 16:32:06.391054 392 external_mini_cluster.cc:1428] Setting key 8342e61b1fe5fa4fe80464e9f887c3fe
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:06.408931 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.392925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.62:42733)
0925 16:32:06.393407 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.393420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.393492 (+ 72us) server_negotiation.cc:407] Connection header received
0925 16:32:06.394122 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.394129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.394458 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.394707 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.395922 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.397461 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.399000 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.399722 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.399905 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.403535 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.403546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.403568 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.403603 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.405522 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.406107 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.406112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.406113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.406223 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.406519 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.406521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.406522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.407028 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.407157 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.407533 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.407823 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
I20260925 16:32:06.408931 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.391645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.0.1:44784)
0925 16:32:06.392391 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.392462 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.393689 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.394765 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.394771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.395240 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.395670 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.395691 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.397712 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.397716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.398806 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.398822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.399943 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.399946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.400058 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.400788 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.400813 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.403304 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.405653 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.405662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.405675 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.405992 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.406307 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.406310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.406312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.406424 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.407136 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.407143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.407282 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.407434 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.407898 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":134,"thread_start_us":213,"threads_started":1}
I20260925 16:32:06.411811 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:06.412990 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.412397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:39102)
0925 16:32:06.412644 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.412816 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 16:32:06.413307 23042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
I20260925 16:32:06.414529 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.414146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:39104)
0925 16:32:06.414372 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.414442 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 16:32:06.414737 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36115: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 16:32:06.421226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36115
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42733,127.0.98.61:36115,127.0.98.60:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 16:32:06.464066 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } attempt: 1
I20260925 16:32:06.465217 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.464874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:39116)
0925 16:32:06.465041 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.465112 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 16:32:06.466542 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.466168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:39122)
0925 16:32:06.466336 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.466440 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 16:32:06.466709 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36115: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
I20260925 16:32:06.534883 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } attempt: 2
I20260925 16:32:06.535960 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.535597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56160)
0925 16:32:06.535779 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.535854 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 16:32:06.536777 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.536525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56164)
0925 16:32:06.536634 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.536695 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 16:32:06.536904 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36115: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
W20260925 16:32:06.557915 23100 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:32:06.558274 23100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:06.558313 23100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:06.558386 23100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:06.563434 23100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:06.563552 23100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:06.563582 23100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:06.563603 23100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:06.563623 23100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:06.571187 23100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42733,127.0.98.61:36115,127.0.98.60:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:06.572886 23100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:06.574543 23100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:06.585601 23106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:06.585592 23108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:06.585592 23105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:06.585815 23100 server_base.cc:1034] running on GCE node
I20260925 16:32:06.586956 23100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:06.588502 23100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:06.589782 23100 hybrid_clock.cc:648] HybridClock initialized: now 1790353926589717 us; error 60 us; skew 500 ppm
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:06.593523 23100 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 16:32:06.594883 23100 webserver.cc:460] Webserver started at http://127.0.98.61:41891/ using document root <none> and password file <none>
I20260925 16:32:06.595584 23100 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:06.595649 23100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:06.595863 23100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:06.598521 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data/instance:
uuid: "d9bb385df20b493eba14db842fd75767"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "108c8e7b12010bd4728a4bc46099312d"
server_key_iv: "dbb2f6264256d6c7972e64868e2b9da8"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.599138 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal/instance:
uuid: "d9bb385df20b493eba14db842fd75767"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "108c8e7b12010bd4728a4bc46099312d"
server_key_iv: "dbb2f6264256d6c7972e64868e2b9da8"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.603842 23100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:06.607362 23115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:06.609089 23100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 16:32:06.609270 23100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal
uuid: "d9bb385df20b493eba14db842fd75767"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "108c8e7b12010bd4728a4bc46099312d"
server_key_iv: "dbb2f6264256d6c7972e64868e2b9da8"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.609406 23100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:06.621959 23100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:06.625857 23100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:06.626070 23100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:06.628088 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } attempt: 3
I20260925 16:32:06.629175 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.628749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56180)
0925 16:32:06.628964 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.629055 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 16:32:06.630242 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.629941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56182)
0925 16:32:06.630083 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.630139 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 16:32:06.630657 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36115: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36115: connect: Connection refused (error 111)
I20260925 16:32:06.634933 23100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36115
I20260925 16:32:06.635310 23167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36115 every 8 connection(s)
I20260925 16:32:06.636234 23100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/data/info.pb
I20260925 16:32:06.639727 23168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:06.640434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 23100
I20260925 16:32:06.640584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-1/wal/instance
I20260925 16:32:06.640930 392 external_mini_cluster.cc:1428] Setting key 3aa6a451382b21fe58a061ee4ab31b07
I20260925 16:32:06.644632 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:06.665129 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.647560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.62:42733)
0925 16:32:06.647779 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.647784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.649106 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 16:32:06.650589 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.650593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.650662 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.650778 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.652799 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.653573 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.655212 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.655618 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.655777 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.659921 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.659925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.659928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.659968 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.661984 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.662869 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.662873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.662875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.663012 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.663578 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.663582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.663583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.663859 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.663977 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.664830 (+ 853us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.664987 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 16:32:06.666486 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.648113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.0.1:34586)
0925 16:32:06.648854 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.648922 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.650250 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.651154 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.651165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.651707 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.652539 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.652580 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.653858 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.653864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.654964 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.654986 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.655916 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.655921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.656180 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.657027 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.657050 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.659532 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.662223 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.662236 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.662269 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.662680 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.663149 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.663153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.663155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.663401 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.663997 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.664006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.664414 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.664667 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.665154 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":203,"threads_started":1}
I20260925 16:32:06.668706 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 16:32:06.674196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task10sHI3/build/debug/bin/kudu
/tmp/dist-test-task10sHI3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45911
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42733,127.0.98.61:36115,127.0.98.60:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:32:06.680642 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.669691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56194)
0925 16:32:06.669985 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.670001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.670110 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.670996 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.671002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.671207 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.671341 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.671348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.672630 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.672635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.673522 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.673539 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.674375 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.674378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.674496 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.675003 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.675016 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.676539 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.679038 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.679041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.679043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.679259 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.679536 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.679539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.679541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.679617 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.680094 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.680098 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.680185 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.680274 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.680438 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":19}
I20260925 16:32:06.680667 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.669942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.0.98.61:36115)
0925 16:32:06.670365 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.670377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.670404 (+ 27us) server_negotiation.cc:407] Connection header received
0925 16:32:06.670457 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.670462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.670709 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.670852 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.671494 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.672432 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.673695 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.674238 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.674356 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.676817 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.676831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.676848 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.676879 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.678908 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.679368 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.679374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.679376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.679456 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.679693 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.679696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.679697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.679986 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.680116 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.680370 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.680501 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":303,"thread_start_us":178,"threads_started":1}
I20260925 16:32:06.683600 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:06.685081 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.684523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59788)
0925 16:32:06.684759 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.684911 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 16:32:06.685431 23116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
I20260925 16:32:06.686690 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.686298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59804)
0925 16:32:06.686475 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.686570 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 16:32:06.687000 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
I20260925 16:32:06.736370 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } attempt: 1
I20260925 16:32:06.737465 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.736959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59818)
0925 16:32:06.737205 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.737324 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 16:32:06.738433 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.738111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59832)
0925 16:32:06.738268 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.738344 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 16:32:06.738611 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
I20260925 16:32:06.773907 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } attempt: 4
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:32:06.786263 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.774804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56204)
0925 16:32:06.775118 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.775141 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.775286 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.775557 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.775560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.775763 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.775885 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.775891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.776684 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.776687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.777480 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.777505 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.778380 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.778385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.778506 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.779255 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.779273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.780957 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.784274 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.784279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.784281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.784615 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.785026 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.785029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.785031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.785130 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.785667 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.785670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.785846 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.785948 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.786119 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260925 16:32:06.786432 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.774865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.0.98.61:36115)
0925 16:32:06.775284 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.775288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.775305 (+ 17us) server_negotiation.cc:407] Connection header received
0925 16:32:06.775380 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.775383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.775433 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.775510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.776009 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.776561 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.777730 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.778206 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.778339 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.781258 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.781271 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.781274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.781317 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.783995 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.784770 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.784775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.784777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.784896 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.785233 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.785236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.785238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.785543 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.785650 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.786049 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.786221 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 16:32:06.787539 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:06.788519 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.788176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59840)
0925 16:32:06.788377 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.788435 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 16:32:06.789408 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.789039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59850)
0925 16:32:06.789160 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.789202 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 16:32:06.789656 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
I20260925 16:32:06.806809 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } attempt: 2
I20260925 16:32:06.807946 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.807506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59866)
0925 16:32:06.807720 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.807817 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 16:32:06.809055 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.808771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59876)
0925 16:32:06.808908 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.808971 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 16:32:06.809276 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
W20260925 16:32:06.819533 23174 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 16:32:06.819932 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 16:32:06.819958 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 16:32:06.820000 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 16:32:06.824431 23174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 16:32:06.824532 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 16:32:06.824551 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 16:32:06.824568 23174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 16:32:06.824584 23174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 16:32:06.830757 23174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42733,127.0.98.61:36115,127.0.98.60:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 15:52:38 UTC on ccycloud.quasar-zbxleh.root.comops.site
build id 4672940
I20260925 16:32:06.832238 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 16:32:06.833621 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 16:32:06.844753 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:06.844753 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 16:32:06.844763 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 16:32:06.844831 23174 server_base.cc:1034] running on GCE node
I20260925 16:32:06.845885 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 16:32:06.847486 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 16:32:06.848702 23174 hybrid_clock.cc:648] HybridClock initialized: now 1790353926848668 us; error 60 us; skew 500 ppm
I20260925 16:32:06.848855 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } attempt: 1
I20260925 16:32:06.849695 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.849385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59888)
0925 16:32:06.849550 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.849613 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 16:32:06.850668 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.850420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59890)
0925 16:32:06.850554 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.850604 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 16:32:06.850792 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45911: connect: Connection refused (error 111)
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 16:32:06.852444 23174 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 16:32:06.853685 23174 webserver.cc:460] Webserver started at http://127.0.98.60:43831/ using document root <none> and password file <none>
I20260925 16:32:06.854423 23174 fs_manager.cc:359] Metadata directory not provided
I20260925 16:32:06.854501 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 16:32:06.854725 23174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 16:32:06.857286 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data/instance:
uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "1584eed1d0e691975039953b0a84143e"
server_key_iv: "068bab1703cb2c5753341fd1f98ba33b"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.857973 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal/instance:
uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "1584eed1d0e691975039953b0a84143e"
server_key_iv: "068bab1703cb2c5753341fd1f98ba33b"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.863306 23174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 16:32:06.867321 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 16:32:06.869122 23174 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 16:32:06.869318 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal
uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77"
format_stamp: "Formatted at 2026-09-25 16:32:06 on dist-test-slave-31kt"
server_key: "1584eed1d0e691975039953b0a84143e"
server_key_iv: "068bab1703cb2c5753341fd1f98ba33b"
server_key_version: "encryptionkey@0"
I20260925 16:32:06.869472 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 16:32:06.882392 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 16:32:06.886139 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 16:32:06.886360 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 16:32:06.895172 23174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45911
I20260925 16:32:06.895454 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45911 every 8 connection(s)
I20260925 16:32:06.896559 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/data/info.pb
I20260925 16:32:06.900511 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 16:32:06.900542 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } attempt: 3
I20260925 16:32:06.903910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task10sHI3/build/debug/bin/kudu as pid 23174
I20260925 16:32:06.904083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task10sHI3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790353612433705-392-0/minicluster-data/master-2/wal/instance
I20260925 16:32:06.904421 392 external_mini_cluster.cc:1428] Setting key 3faec4fbfaccbbbd7a13bf1120ae3e14
I20260925 16:32:06.907248 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:32:06.916864 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.901194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59904)
0925 16:32:06.901419 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.901445 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.901621 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.904379 (+ 2758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.904383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.904638 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.904778 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.904786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.906541 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.906544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.907243 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.907254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.908338 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.908342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.908438 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.909303 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.909321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.911171 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.914133 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.914137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.914140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.914497 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.914979 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.914982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.914984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.915071 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.916172 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.916178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.916353 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.916479 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.916665 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:06.917953 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } attempt: 2
I20260925 16:32:06.918467 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.905166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.0.98.60:45911)
0925 16:32:06.905578 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.905588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.905607 (+ 19us) server_negotiation.cc:407] Connection header received
0925 16:32:06.905781 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.905784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.905830 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.905926 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.906879 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.907574 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.908593 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.908873 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.908963 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.911765 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.911769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.911771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.911797 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.914058 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.914639 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.914647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.914655 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.914815 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.915180 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.915184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.915186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.915974 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.916171 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.916638 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.917021 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":342,"spinlock_wait_cycles":7296,"thread_start_us":196,"threads_started":1}
I20260925 16:32:06.918606 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.902119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.0.98.60:45911)
0925 16:32:06.902629 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.902649 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.902695 (+ 46us) server_negotiation.cc:407] Connection header received
0925 16:32:06.903507 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.903528 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.903998 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.904430 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.904938 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.906419 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.907412 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.908183 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.908362 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.911377 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.911383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.911397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.911430 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.913989 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.914770 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.914773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.914774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.914854 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.915180 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.915184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.915186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.915974 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.916114 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.917145 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.917306 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":338,"spinlock_wait_cycles":9984,"thread_start_us":163,"threads_started":1}
I20260925 16:32:06.922030 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.909855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.0.1:34592)
0925 16:32:06.910443 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.910508 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.910731 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.911179 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.911185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.911757 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.911950 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.911960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.912976 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.912980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.913984 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.913996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.914790 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.914794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.914922 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.915432 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.915444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.917746 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.920125 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.920131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.920136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.920425 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.920791 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.920794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.920795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.920882 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.921316 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.921323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.921454 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.921599 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.921835 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":149,"threads_started":1}
I20260925 16:32:06.922691 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.909943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.98.62:42733)
0925 16:32:06.910088 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.910093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.910606 (+ 513us) server_negotiation.cc:407] Connection header received
0925 16:32:06.910889 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.910893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.910975 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.911079 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.912102 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.912839 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.914165 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.914547 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.914655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.917973 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.917977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.917980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.918020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.920020 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.920575 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.920580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.920582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.920691 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.920983 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.921000 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.921002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.921212 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.921306 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.922404 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.922544 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 16:32:06.923951 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 16:32:06.924803 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: Bootstrap starting.
I20260925 16:32:06.928498 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.918520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59924)
0925 16:32:06.918738 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.918752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.918828 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.919139 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.919142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.920156 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.920249 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.920255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.921239 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.921243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.921967 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.921978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.922793 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.922797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.922876 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.923356 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.923370 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.924987 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.926857 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.926860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.926863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.927124 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.927428 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.927431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.927432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.927522 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.927991 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.927993 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.928084 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.928165 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.928310 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260925 16:32:06.928951 23168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:06.929188 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.918637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.0.98.60:45911)
0925 16:32:06.918898 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.918903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.918916 (+ 13us) server_negotiation.cc:407] Connection header received
0925 16:32:06.918949 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.918952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.918995 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.919521 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.920356 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.921107 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.922124 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.922698 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.922799 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.925121 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.925124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.925128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.925152 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.926743 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.927239 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.927242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.927244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.927337 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.927646 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.927650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.927652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.927887 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.928820 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.928896 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.929026 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":152,"threads_started":1}
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 16:32:06.932981 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28: Bootstrap starting.
I20260925 16:32:06.934839 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.924436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36115 (local address 127.0.0.1:56214)
0925 16:32:06.924582 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.924599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.924680 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.925090 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.925119 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.925301 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.925437 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.925444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.926649 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.926652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.927460 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.927470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.928143 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.928146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.928246 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.928959 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.928975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.930846 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.933024 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.933028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.933031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.933341 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.933658 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.933661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.933663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.933750 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.934291 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.934294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.934396 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.934528 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.934700 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 16:32:06.935027 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.924721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.0.98.61:36115)
0925 16:32:06.924842 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.924846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.924860 (+ 14us) server_negotiation.cc:407] Connection header received
0925 16:32:06.924906 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.924909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.924960 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.925058 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.925575 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.926499 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.927588 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.927894 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.927979 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.931107 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.931121 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.931126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.931168 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.932905 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.933465 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.933469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.933471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.933572 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.933861 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.933865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.933866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.934195 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.934291 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.934651 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.934831 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 16:32:06.935811 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 16:32:06.936614 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28: Neither blocks nor log segments found. Creating new log.
I20260925 16:32:06.937744 23093 log.cc:824] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:06.938081 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.922415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.0.98.62:42733)
0925 16:32:06.922701 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.922712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.926378 (+ 3666us) server_negotiation.cc:407] Connection header received
0925 16:32:06.926418 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.926421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.926469 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.926547 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.927415 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.928875 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.929902 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.930198 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.930289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.933554 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.933559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.933562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.933592 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.935891 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.936515 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.936519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.936522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.936615 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.936926 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.936930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.936932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.937195 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.937387 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.937694 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.937878 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260925 16:32:06.938862 23168 log.cc:824] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:06.941959 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28: No bootstrap required, opened a new log
I20260925 16:32:06.941968 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: No bootstrap required, opened a new log
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 16:32:06 dist-test-slave-31kt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790353926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 16:32:06.946695 23168 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.946695 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.946988 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1f94304e7a4af5b42921953ec11c28, State: Initialized, Role: FOLLOWER
I20260925 16:32:06.947000 23168 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9bb385df20b493eba14db842fd75767, State: Initialized, Role: FOLLOWER
I20260925 16:32:06.947403 23168 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.947403 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.947826 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.936304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45911 (local address 127.0.0.1:59934)
0925 16:32:06.936412 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 16:32:06.936425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 16:32:06.936503 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 16:32:06.936696 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 16:32:06.936698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 16:32:06.936884 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 16:32:06.936952 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.936957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.937802 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.937805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.938643 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 16:32:06.938653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.939553 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.939557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 16:32:06.939634 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.941927 (+ 2293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 16:32:06.941944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 16:32:06.943723 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 16:32:06.945889 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.945892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.945895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.946144 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.946407 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 16:32:06.946409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 16:32:06.946411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 16:32:06.946482 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 16:32:06.946904 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 16:32:06.946907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 16:32:06.946982 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 16:32:06.947075 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 16:32:06.947431 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 16:32:06.948419 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.936434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.0.98.60:45911)
0925 16:32:06.936518 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.936522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.936545 (+ 23us) server_negotiation.cc:407] Connection header received
0925 16:32:06.936576 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.936578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.936615 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.936735 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.937050 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.937660 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.938796 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.939305 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.939417 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.943891 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.943895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.943898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.943920 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.945721 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.946246 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.946249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.946251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.946326 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.946565 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.946567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.946569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.946797 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.946874 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.948186 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.948294 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 16:32:06.949195 23168 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:06.949317 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:06.950838 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.939527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.98.61:36115)
0925 16:32:06.939662 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.939668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.939781 (+ 113us) server_negotiation.cc:407] Connection header received
0925 16:32:06.940468 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.940472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.940532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.940624 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.941290 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.942113 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.943294 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.943722 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.943809 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.946848 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.946852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.946855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.946885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.948365 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.949145 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.949149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.949150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.949255 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.949681 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.949684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.949686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.949969 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.950183 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.950547 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.950676 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":52}
I20260925 16:32:06.952064 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: Bootstrap starting.
I20260925 16:32:06.952903 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:06.953078 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:06.956326 23243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: Neither blocks nor log segments found. Creating new log.
W20260925 16:32:06.956547 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:06.956681 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 16:32:06.957073 23276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:06.957172 23276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:32:06.957206 23243 log.cc:824] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: Log is configured to *not* fsync() on all Append() calls
I20260925 16:32:06.957450 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:06.957612 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:06.958108 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 16:32:06.959568 23067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6a1f94304e7a4af5b42921953ec11c28"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34602
I20260925 16:32:06.959817 23067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 16:32:06.959939 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:06.961597 23067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.962831 23067 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [CANDIDATE]: Term 1 election: Requested vote from peers d9bb385df20b493eba14db842fd75767 (127.0.98.61:36115), f4bbe55fe32e4e2fbb8c6e13d566ad77 (127.0.98.60:45911)
I20260925 16:32:06.963919 23142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a1f94304e7a4af5b42921953ec11c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d9bb385df20b493eba14db842fd75767"
I20260925 16:32:06.964278 23142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:06.965948 23142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1f94304e7a4af5b42921953ec11c28 in term 1.
I20260925 16:32:06.966764 23042 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1f94304e7a4af5b42921953ec11c28, d9bb385df20b493eba14db842fd75767; no voters:
I20260925 16:32:06.967198 23253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 16:32:06.967525 23253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1f94304e7a4af5b42921953ec11c28, State: Running, Role: LEADER
I20260925 16:32:06.967983 23253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.970449 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1f94304e7a4af5b42921953ec11c28. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:06.970628 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:06.965600 23025 master_runner.cc:186] Error getting master registration for 127.0.98.60:45911: OK, instance_id { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" instance_seqno: 1790353926892520 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:32:06.971704 23025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:32:06.969635 23100 master_runner.cc:186] Error getting master registration for 127.0.98.60:45911: OK, instance_id { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" instance_seqno: 1790353926892520 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 16:32:06.973994 23100 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 16:32:06.975685 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: No bootstrap required, opened a new log
I20260925 16:32:06.982031 23243 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.982429 23243 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bbe55fe32e4e2fbb8c6e13d566ad77, State: Initialized, Role: FOLLOWER
I20260925 16:32:06.982906 23243 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } }
I20260925 16:32:06.984839 23243 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 16:32:06.985003 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.972193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.98.62:42733)
0925 16:32:06.972329 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.972334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.974575 (+ 2241us) server_negotiation.cc:407] Connection header received
0925 16:32:06.974794 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.974797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.974855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.974910 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.975980 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.976788 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.978446 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.978834 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.978917 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.981197 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.981205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.981208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.981244 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.983013 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.983551 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.983555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.983557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.983645 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.983914 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.983917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.983919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.984180 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.984288 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.984652 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.984807 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 16:32:06.984915 23217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a1f94304e7a4af5b42921953ec11c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77"
I20260925 16:32:06.985299 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 16:32:06.987438 23217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1f94304e7a4af5b42921953ec11c28 in term 1.
I20260925 16:32:06.989420 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.973112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.0.98.61:36115)
0925 16:32:06.973265 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.973271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.974721 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 16:32:06.974940 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.974943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.975013 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.975064 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.976214 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.976993 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.978244 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.978529 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.978596 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.980597 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.980602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.980605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.980639 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.983531 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.984388 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.984392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.984394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.984485 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.988097 (+ 3612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.988102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.988104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.988399 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.988540 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.989026 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.989221 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 16:32:06.990170 23286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 16:32:06.989997 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:06.990928 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:06.991360 23286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 16:32:06.993086 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:06.973875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.0.98.60:45911)
0925 16:32:06.974039 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:06.974045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:06.978292 (+ 4247us) server_negotiation.cc:407] Connection header received
0925 16:32:06.978409 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:06.978413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:06.978474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:06.978560 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:06.981682 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.982544 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.986896 (+ 4352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:06.987205 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:06.987285 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:06.989043 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 16:32:06.989047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 16:32:06.989050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 16:32:06.989078 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 16:32:06.990678 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.991521 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.991526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.991528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.991643 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 16:32:06.991985 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 16:32:06.991989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 16:32:06.991990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 16:32:06.992254 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 16:32:06.992393 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:06.992756 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 16:32:06.992905 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 16:32:07.005946 23301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 16:32:07.006110 23301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 16:32:07.012080 23217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [term 1 FOLLOWER]: Refusing update from remote peer 6a1f94304e7a4af5b42921953ec11c28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:07.015628 23280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:07.022081 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1f94304e7a4af5b42921953ec11c28. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:07.022289 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:07.022949 23142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [term 1 FOLLOWER]: Refusing update from remote peer 6a1f94304e7a4af5b42921953ec11c28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 16:32:07.026620 23280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 16:32:07.031620 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1f94304e7a4af5b42921953ec11c28. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:07.031814 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:07.035295 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:07.035493 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:07.037120 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:07.037314 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:07.039209 23303 mvcc.cc:205] Tried to move back new op lower bound from 7333289685026926592 to 7333289684865417216. Current Snapshot: MvccSnapshot[applied={T|T < 7333289685026926592}]
I20260925 16:32:07.039448 23304 mvcc.cc:205] Tried to move back new op lower bound from 7333289685026926592 to 7333289684865417216. Current Snapshot: MvccSnapshot[applied={T|T < 7333289685026926592}]
I20260925 16:32:07.044236 23286 catalog_manager.cc:1347] Generated new cluster ID: f26b9f7ea4a94fbd8f7fb29199a781b9
I20260925 16:32:07.044332 23286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 16:32:07.050875 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:07.051045 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28 [sys.catalog]: This master's current role is: LEADER
I20260925 16:32:07.052489 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a1f94304e7a4af5b42921953ec11c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1f94304e7a4af5b42921953ec11c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } peers { permanent_uuid: "d9bb385df20b493eba14db842fd75767" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36115 } } peers { permanent_uuid: "f4bbe55fe32e4e2fbb8c6e13d566ad77" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45911 } } }
I20260925 16:32:07.052651 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 16:32:07.054105 23306 mvcc.cc:205] Tried to move back new op lower bound from 7333289685026926592 to 7333289684865417216. Current Snapshot: MvccSnapshot[applied={T|T < 7333289685026926592}]
I20260925 16:32:07.074029 23286 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 16:32:07.078558 23286 catalog_manager.cc:1504] Loading token signing keys...
I20260925 16:32:07.096423 23286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1f94304e7a4af5b42921953ec11c28: Generated new TSK 0
I20260925 16:32:07.097589 23286 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 16:32:07.133728 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.131327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.0.98.62:42733)
0925 16:32:07.131576 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.131583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.132525 (+ 942us) server_negotiation.cc:407] Connection header received
0925 16:32:07.132739 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.132743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.132804 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.132860 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.133568 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34620: BlockingRecv error: recv got EOF from 127.0.0.1:34620 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 16:32:07.135315 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.132273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.0.98.61:36115)
0925 16:32:07.132578 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.132584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.134336 (+ 1752us) server_negotiation.cc:407] Connection header received
0925 16:32:07.134520 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.134523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.134579 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.134632 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.135168 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56234: BlockingRecv error: recv got EOF from 127.0.0.1:56234 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
W20260925 16:32:07.137312 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.135936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.0.98.60:45911)
0925 16:32:07.136113 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.136118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.136138 (+ 20us) server_negotiation.cc:407] Connection header received
0925 16:32:07.136207 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.136211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.136274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.136408 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.137133 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59944: BlockingRecv error: recv got EOF from 127.0.0.1:59944 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 16:32:07.143901 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.141671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.98.61:36115)
0925 16:32:07.141843 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.141848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.142517 (+ 669us) server_negotiation.cc:407] Connection header received
0925 16:32:07.142807 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.142811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.142915 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.143018 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.143738 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56246: BlockingRecv error: recv got EOF from 127.0.0.1:56246 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 16:32:07.151166 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.148834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.98.61:36115)
0925 16:32:07.149004 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.149009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.149830 (+ 821us) server_negotiation.cc:407] Connection header received
0925 16:32:07.150131 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.150136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.150211 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.150279 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.151006 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56260: BlockingRecv error: recv got EOF from 127.0.0.1:56260 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 16:32:07.158913 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.156650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.0.98.61:36115)
0925 16:32:07.156805 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.156811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.157541 (+ 730us) server_negotiation.cc:407] Connection header received
0925 16:32:07.157875 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.157879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.157948 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.158025 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.158751 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56276: BlockingRecv error: recv got EOF from 127.0.0.1:56276 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 16:32:07.168783 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.166544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.0.98.61:36115)
0925 16:32:07.166823 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.166828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.167560 (+ 732us) server_negotiation.cc:407] Connection header received
0925 16:32:07.167788 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.167792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.167856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.167912 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.168625 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56292: BlockingRecv error: recv got EOF from 127.0.0.1:56292 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
W20260925 16:32:07.183159 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.180921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.0.98.61:36115)
0925 16:32:07.181115 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.181121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.181913 (+ 792us) server_negotiation.cc:407] Connection header received
0925 16:32:07.182168 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.182173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.182239 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.182299 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.182989 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56298: BlockingRecv error: recv got EOF from 127.0.0.1:56298 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 16:32:07.204182 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.202730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.0.98.61:36115)
0925 16:32:07.202894 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.202900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.203179 (+ 279us) server_negotiation.cc:407] Connection header received
0925 16:32:07.203347 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.203349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.203407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.203505 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.204021 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56300: BlockingRecv error: recv got EOF from 127.0.0.1:56300 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 16:32:07.240307 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.238952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.0.98.61:36115)
0925 16:32:07.239190 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.239195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.239271 (+ 76us) server_negotiation.cc:407] Connection header received
0925 16:32:07.239431 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.239434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.239494 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.239579 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.240160 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56314: BlockingRecv error: recv got EOF from 127.0.0.1:56314 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260925 16:32:07.308470 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.307195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.0.98.61:36115)
0925 16:32:07.307435 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.307441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.307580 (+ 139us) server_negotiation.cc:407] Connection header received
0925 16:32:07.307753 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.307756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.307804 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.307877 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.308360 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56322: BlockingRecv error: recv got EOF from 127.0.0.1:56322 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260925 16:32:07.440991 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.439553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.0.98.61:36115)
0925 16:32:07.439693 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.439696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.439838 (+ 142us) server_negotiation.cc:407] Connection header received
0925 16:32:07.439936 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.439938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.440021 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.440125 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.440837 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56332: BlockingRecv error: recv got EOF from 127.0.0.1:56332 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 16:32:07.701058 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 16:32:07.699784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.98.61:36115)
0925 16:32:07.699969 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:07.699973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:07.700068 (+ 95us) server_negotiation.cc:407] Connection header received
0925 16:32:07.700222 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:07.700225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:07.700283 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:07.700371 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 16:32:07.700926 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56338: BlockingRecv error: recv got EOF from 127.0.0.1:56338 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 16:32:07.958248 23275 catalog_manager.cc:1259] Loaded cluster ID: f26b9f7ea4a94fbd8f7fb29199a781b9
I20260925 16:32:07.958345 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: loading cluster ID for follower catalog manager: success
I20260925 16:32:07.960028 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: acquiring CA information for follower catalog manager: success
I20260925 16:32:07.961274 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9bb385df20b493eba14db842fd75767: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:32:08.007536 23301 catalog_manager.cc:1259] Loaded cluster ID: f26b9f7ea4a94fbd8f7fb29199a781b9
I20260925 16:32:08.007634 23301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: loading cluster ID for follower catalog manager: success
I20260925 16:32:08.010123 23301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: acquiring CA information for follower catalog manager: success
I20260925 16:32:08.011626 23301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4bbe55fe32e4e2fbb8c6e13d566ad77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 16:32:08.220558 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:08.215958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.98.61:36115)
0925 16:32:08.216245 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:08.216250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:08.216384 (+ 134us) server_negotiation.cc:407] Connection header received
0925 16:32:08.216566 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:08.216570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:08.216748 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:08.216861 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:08.217136 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.217736 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.218693 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.218936 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.219030 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:08.219413 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:08.220049 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:08.220176 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:08.220264 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 16:32:08.220349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260925 16:32:08.232939 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:08.225560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.0.98.60:45911)
0925 16:32:08.225891 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:08.225896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:08.226457 (+ 561us) server_negotiation.cc:407] Connection header received
0925 16:32:08.226634 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:08.226639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:08.226897 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:08.227095 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:08.227657 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.228562 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.229980 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.230406 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.230555 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:08.231156 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:08.232013 (+ 857us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:08.232173 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:08.232685 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 16:32:08.232782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":89,"threads_started":1}
I20260925 16:32:08.241881 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:08.237211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.0.98.61:36115)
0925 16:32:08.237389 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:08.237395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:08.238151 (+ 756us) server_negotiation.cc:407] Connection header received
0925 16:32:08.238371 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:08.238375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:08.238443 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:08.238522 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:08.238785 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.239512 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.240567 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.240888 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.240997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:08.241339 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:08.241453 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:08.241524 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:08.241610 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 16:32:08.241684 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 16:32:08.242466 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:08.237319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.0.98.60:45911)
0925 16:32:08.237512 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:08.237517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:08.237612 (+ 95us) server_negotiation.cc:407] Connection header received
0925 16:32:08.237932 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:08.237936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:08.238008 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:08.238118 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:08.238436 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.239253 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.240679 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.241041 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.241162 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:08.241436 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:08.241570 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:08.241643 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:08.242203 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 16:32:08.242289 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 16:32:08.243506 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 16:32:08.237544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.0.98.62:42733)
0925 16:32:08.237894 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 16:32:08.237899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 16:32:08.238160 (+ 261us) server_negotiation.cc:407] Connection header received
0925 16:32:08.238491 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 16:32:08.238495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 16:32:08.238778 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 16:32:08.239031 (+ 253us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 16:32:08.239350 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.240234 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.241513 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 16:32:08.241917 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 16:32:08.242029 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 16:32:08.242472 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 16:32:08.242998 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 16:32:08.243095 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 16:32:08.243241 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 16:32:08.243326 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260925 16:32:08.248558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 23025
I20260925 16:32:08.255610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 23100
I20260925 16:32:08.263068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task10sHI3/build/debug/bin/kudu with pid 23174
2026-09-25T16:32:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3219 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6577 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (315820 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST